:W&Bá¾·L×ÁM okrvw1yuª; 9 /0.29.0+7872fd9d9f86ca5c50617e19cb168122b7f959480.70.0ÙÐÿ®û‚" 3e45iy1qc25n okrvw1yuŠ× okrvw1yujuju11-uc-san-diegoSPAtop"š _wandb‚{}  batch_size‚256  dataset_limit‚0.2  dropout‚0.16025997468135922 epochs‚10  hidden_dim‚128 ' learning_rate‚0.0010523537353772008  num_attention_heads‚8  num_branch_encoder_layers‚7  num_embedding_layers‚14  num_encoder_layers‚6Bjjtttt_blocks_okrvw1yubs2s2ev0zj*jj-spatop-sweepagent-tttt-pairwise-1-8pwgvŠ ÅéÔ¶Çåš3 (H¨¸hp"3.12.10*0.29.0j linux-x86_64à#&7 okrvw1yu&  _runtime‚0  _wandb.runtime‚0r ˆjÒä -Linux-6.8.0-137-generic-x86_64-with-glibc2.36CPython 3.12.10 ÄéÔ¨çêÓ*-of*=/data/spatop/options_files/sweep_tttt_jj/config_okrvw1yu.json*-l* /scratch/logs*-n*jjtttt_blocks_okrvw1yu2-m spanet.trainRjacksonjulian2014@gmail.comZ /scratch/logsb*jj-spatop-sweepagent-tttt-pairwise-1-8pwgvr/opt/conda/bin/pythonº t7whm7np441x1b09c5wvivlffdrcc7srÈÃöÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€øáo€€ëæÄ]ª€àäêÞÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-c1fdb213-5a16-84e3-13b4-cf29bc35dbf8Ê13.2º t7whm7np441x1b09c5wvivlffdrcc7srµª½Ç2  requirements.txt}•…ÒW okrvw1yuZF (H¨¸ (H¨¸ (hpð"3.12.10*0.29.0j linux-x86_64y¬ÜªY okrvw1yuZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.29.0j linux-x86_64ebþ«*  okrvw1yub trainer/global_step2•ÓÃU\  okrvw1yuZK (H¨¸ (H¨¸(8hpð"3.12.10*0.29.0j linux-x86_64³’€/  okrvw1yub*"trainer/global_step2Œ|‚K  okrvw1yuj: ÆéÔÀ¥Æ *LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] 'è>>“  okrvw1yuj  ÆéÔ¨¼ÀÇî â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 866 K │ train │ 0 │ │ 1 │ encoder │ JetEncoderWithPairwise │ 2.0 M │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 53.9 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 48 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ š°Ø\ okrvw1yujÊ ÆéÔÀ„ÍÈ·Trainable params: 56.7 M Non-trainable params: 86 Total params: 56.7 M Total estimated model params size (MB): 226.908 Modules in train mode: 5928 Modules in eval mode: 0 Total FLOPs: 0 ¡#™%q okrvw1yuj` ÆéÔØ«çÈPwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). ¯µÚ{ okrvw1yujj ÆéÔ€àíÈZwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). oáxNy okrvw1yujh ÆéÔ°¾óÈXwandb: WARNING Config item 'num_encoder_layers' was locked by 'sweep' (ignored update). JüWÕ€ okrvw1yujo ÆéÔ¨ðÿÈ_wandb: WARNING Config item 'num_branch_encoder_layers' was locked by 'sweep' (ignored update). xî+ªz okrvw1yuji ÆéÔˆ¼†ÉYwandb: WARNING Config item 'num_attention_heads' was locked by 'sweep' (ignored update). ªm˜t okrvw1yujc ÆéÔ芌ÉSwandb: WARNING Config item 'dataset_limit' was locked by 'sweep' (ignored update). U,ÊÂq okrvw1yuj` ÆéÔ˜ã“ÉPwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). qf%t okrvw1yujc ÆéÔ¨ŸÉSwandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). Hêëýn okrvw1yuj] ÆéÔ¨þ¤ÉMwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). vY¸Gm okrvw1yuj\ ÆéÔ ¿¬ÉLwandb: WARNING Config item 'epochs' was locked by 'sweep' (ignored update). ;ì„dÜ okrvw1yu*Ê  transformer_dim‚128  transformer_dim_scale‚2.0  initial_embedding_dim‚128  position_embedding_dim‚128 ! num_branch_embedding_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚1  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚"all"  pairwise_block_embeddings‚1  pairwise_cross_type‚true  pairwise_feature_set‚""  pairwise_eq2to2‚false  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 F event_info_file‚2"/data/spatop/event_files/tttt/tttt_hadronic.yaml" = training_file‚+"/data/spatop/tttt_15M/h5/tttt_training.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"S‰b‡ë okrvw1yujÙ ÆéÔø¨‡ÐÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. qÀzðN okrvw1yuj¼ ÆéÔØ֗Ы/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. ý„Qàë okrvw1yujÙ ÉéÔ ž×èÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. –ŽXP okrvw1yuj¾ ÉéÔð€æè­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. f•’‚(:ˆ ÕéÔ—˜²k gpu.0.gpu‚10.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚38.0 gpu.0.powerWatts‚177.411' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.35275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ“©î‚(:ä ÕéÔ°›°o disk.sdb.in‚0 disk.sdb.out‚ 1.6640625# proc.memory.rssMB‚ 2904.80078125 disk.sda1.out‚0 disk.md0p1.in‚0 disk.sda2.out‚ 1.6640625 disk.nvme1n1.in‚ 0.05078125 network.sent‚33736 disk.rbd0.in‚0 disk.md0p1.out‚1.609375 disk.nvme1n1.out‚ 66.3515625 disk.md0.in‚0 disk.rbd2.in‚0 disk.sda.in‚0( proc.memory.availableMB‚ 5364.3203125$ disk./.usageGB‚2993.235122680664 disk.rbd2.out‚0* proc.memory.percent‚17.729496955871582 cpu‚24.669220583990572 proc.cpu.threads‚17) disk./.usagePercent‚83.69210679637497 disk.sda2.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 0.02734375 disk.nvme0n1.out‚ 0.50390625$ memory_percent‚67.25878715515137 disk.dm-0.out‚ 66.85546875 disk.md0.out‚1.609375 disk.sdb2.in‚0 disk.sdb2.out‚ 1.6640625 disk.sda.out‚ 1.6640625 network.recv‚38808 disk.rbd0.out‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚0.078125ˆrûù ‚(:ï äéÔ“¿„m disk.md0.out‚3.375 cpu‚24.756588522952104 network.sent‚52776 disk.sda2.in‚0 disk.rbd2.out‚0 disk.sda1.out‚0 disk.sdb1.out‚0 disk.dm-0.in‚0.078125 disk.nvme0n1.in‚ 0.02734375 disk.sda.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2905.19140625 disk.rbd2.in‚0 disk.sdb2.out‚ 3.48095703125$ memory_percent‚67.26701259613037 disk.md0.in‚0 disk.sdb2.in‚0 disk.sda.out‚ 3.48095703125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sdb1.in‚0$ disk./.usageGB‚2993.485122680664 disk.sdb.out‚ 3.48095703125 disk.nvme1n1.in‚ 0.05078125) proc.memory.availableMB‚ 5362.97265625 network.recv‚57012( disk./.usagePercent‚83.6990969010081 disk.md0p1.in‚0 disk.md0p1.out‚3.375* proc.memory.percent‚17.731881141662598 disk.sda2.out‚ 3.48095703125 disk.nvme0n1.out‚ 0.55078125! disk.nvme1n1.out‚ 196.58984375 disk.dm-0.out‚ 197.140625 proc.cpu.threads‚17 disk.sdb.in‚0Žgôú’!‚(:ˆ äéÔêåªm gpu.0.gpu‚36.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚37.0 gpu.0.powerWatts‚119.701' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.92525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"——Í‘"‚(:‡ óéÔצ£m gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚39.0 gpu.0.powerWatts‚179.246' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.8115 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 )q}ü#‚(:ò óéÔßÓŒn disk.sdb1.out‚0 disk.sda2.in‚0 disk.nvme1n1.out‚ 262.2734375 disk.md0.in‚0) proc.memory.availableMB‚ 5361.96484375 network.sent‚63870 disk.sdb.out‚ 4.8720703125 network.recv‚74821 disk.md0p1.out‚ 4.71484375 disk.dm-0.in‚0.078125 disk.rbd2.in‚0 proc.memory.rssMB‚ 2905.28125) proc.memory.percent‚17.73242950439453 cpu‚24.81484177675268$ disk./.usageGB‚2993.485176086426 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.out‚ 4.8720703125 disk.nvme1n1.in‚ 0.05078125 proc.cpu.threads‚17 disk.sdb2.in‚0 disk.rbd0.out‚0 disk.md0p1.in‚0 disk.sdb.in‚0 disk.rbd2.out‚0) disk./.usagePercent‚83.69909839425556 disk.sdb2.out‚ 4.8720703125 disk.sda.out‚ 4.8720703125 disk.sda1.in‚0 disk.nvme0n1.in‚ 0.02734375 disk.nvme0n1.out‚ 1.04296875 disk.md0.out‚ 4.71484375 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.dm-0.out‚ 263.31640625$ memory_percent‚67.273163795471193c€Ï3$b/ lr-AdamW/pg1"trainer/global_step2:JP˜‰7&3%b/ lr-AdamW/pg2"trainer/global_step2:JPÂ}-&) _wandbruntime‚54  _runtime‚54Qu¡ä¿'º _runtime‚ 54.837870937 _step‚0 & lr-AdamW/pg1‚4.265122666127613e-05 & lr-AdamW/pg2‚4.265122666127613e-05 trainer/global_step‚49 $ _timestamp‚1.788495997060411e+09¨ˆÔŽÁ(¼ _runtime‚ 54.837870937 _step‚0 & lr-AdamW/pg1‚4.265122666127613e-05 & lr-AdamW/pg2‚4.265122666127613e-05 trainer/global_step‚49 $ _timestamp‚1.788495997060411e+09ÞI•a)b] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JPSîün*bj GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current"trainer/global_step2:JPªõB™i+be BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JP¢VHe,ba >DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JP‰ÁHúm-bi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JPíÛ#vn.bj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JPÊI]n/bj GDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated"trainer/global_step2:JP¾é`r0bn KDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JPYµ(tf1bb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JPƒX™^2bZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JP”,'Ÿf3bb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JPË ?÷h4bd ADeviceStatsMonitor.on_train_batch_start/active.small_pool.current"trainer/global_step2:JP“°mzl5bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JPÑoÛžj6bf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JPƒ<‰ªq7bm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JPÈk©Sr8bn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPÏÛr~i9be BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JPÐÉ) l:bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JP.Sèâx;bt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPx8ákba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JP¾.&‘g?bc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JP €—km@bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JP1âŠúkAbg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JPÉ«ÞnBbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JP¹‘®ugCbc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JP ›«bDb^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JPf+0nEbj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPW¦+¡sFbo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPûˆþfGbb ?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed"trainer/global_step2:JP°ÎÔŠlHbh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JPV^™×rIbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JPÓä€fJbb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JP“žuífKbb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JP•xM_Lb[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JP½Ã.ÏdMb` =DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak"trainer/global_step2:JPê«®ÒiNbe BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JPÙIíònObj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JP¸¿þEtPbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPëÀsxpQbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JP£kRbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JP)9iÑlSbh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JPô3PjTbf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JP˜¼¨€hUbd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JPžŠ·šrVbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JP)oÿjWbf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JPBC×]XbY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JPŒ«µ _Yb[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JPêƒ7^gZbc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JPPRb[b^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JP‰ùs\bo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JPÉžX³n]bj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JP¼GZÕe^ba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JP·²an_bj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JP>"–6o`bk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JPn=”eWabS 0DeviceStatsMonitor.on_train_batch_start/num_ooms"trainer/global_step2:JP«/`Mhbbd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JP[ Jþkcbg DDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JPR×¾ªkdbg DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated"trainer/global_step2:JPÞø®òfebb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JP^äcfb_ DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JP×áNobk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JP>Ñ—p€bk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JP¼úÞ—qbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JP­Ü½_‚bZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JP°¾fõrƒbm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JPÞ‡cXo„bj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JP<7AÛ`…b[ 8DeviceStatsMonitor.on_train_batch_start/segment.all.peak"trainer/global_step2:JPÆ0÷pk†bf CDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated"trainer/global_step2:JPúSnq‡bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JP‘ëódˆb_ DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JPjÿ°ng bb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JPñPo£bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JPàˆƒ¬.¤) _wandbruntime‚54  _runtime‚54Ñ0_‰ù¥™O WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 4161441198080 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚900558 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 59379435008 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 3972714343024 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 55510146804 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 56630104576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408965 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 4104811093504 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 4591679035392 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235840512 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 trainer/global_step‚49 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1186 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚1 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720837 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚614 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30506 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4591419254784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1018794 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚610300 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30749 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1018792 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 55720900096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚290258 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 4104793090560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 4532299600384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134970368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30749 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚610298 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 4029149496644 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1235123 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2253915 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134970368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚289646 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 4532275660288 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189000192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1235123 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 QCDeviceStatsMonitor.on_train_batch_start/reí7; ¦quested_bytes.all.current‚ 925006816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 56418114260 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1259969 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259780608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 4160513990656 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 4104793090560 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 55720900096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚612 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1018794 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 56630104576 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1259969 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259780608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 4160513990656 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 4104793090560 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 55720900096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚612 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1018794 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 56630104576 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1186 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚1 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720837 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚614 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30506 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4591419254784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1018794 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚610300 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30749 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1018792 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 55720900096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚290258 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 4104793090560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 4532299600384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569Bd|†`§b[ 8DeviceStatsMonitor.on_train_batch_end/active.all.current"trainer/global_step2:JPWŸàIo¨bj GDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current"trainer/global_step2:JP®a=Õp©bk HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JPµf,†^ªbY 6DeviceStatsMonitor.on_train_batch_end/segment.all.peak"trainer/global_step2:JP‰þªÇl«bg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JP„¿G]¬bX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JPn5h2f­ba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JPÁþúVr®bm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JP=ö~Çi¯bd ADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed"trainer/global_step2:JPD}:Œu°bp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JPè¹`$V±bQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JPÀRßNi²bd ADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current"trainer/global_step2:JPú:œf³ba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP2ˆm´bh EDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JP_4Fîkµbf CDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JPÉ’ü¼]¶bX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JP_33h·bc @DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated"trainer/global_step2:JP%\že¸b` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JPrK±ãk¹bf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JPû»oºbj GDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current"trainer/global_step2:JPùyl»bg DDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JPO¸Ä`g¼bb ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JP@kCb½b] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JP¶xr¾bm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JP$ø§p¿bk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JPÖ!ÎqÀbl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPuÕàUwÁbr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP›òü›sÂbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP55×^ÃbY 6DeviceStatsMonitor.on_train_batch_end/active.all.freed"trainer/global_step2:JPH½ÀmÄbh EDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated"trainer/global_step2:JP¢lÅbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JPÔ£®hÆbc @DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current"trainer/global_step2:JPñ_DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JPI97rÏbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP)*èmÐbh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JP‘ŒAhÑbc @DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current"trainer/global_step2:JPÉk\]nÒbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JPéT½8mÓbh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPß,«íbÔb] :DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams"trainer/global_step2:JPÚHŸiÕbd ADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated"trainer/global_step2:JPÌSŠkÖbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JP*œFvf×ba >DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JP †5ülØbg DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated"trainer/global_step2:JPS}ˆfÙba >DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JP÷2g¢\ÚbW 4DeviceStatsMonitor.on_train_batch_end/max_split_size"trainer/global_step2:JP\;Ã7_ÛbZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JP²ñ°‹uÜbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JPqâƒúcÝb^ ;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak"trainer/global_step2:JPÙžyµ^ÞbY 6DeviceStatsMonitor.on_train_batch_end/num_device_alloc"trainer/global_step2:JPGêùOeßb` =DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak"trainer/global_step2:JP¼åYyeàb` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JPˆ£_mábh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPkhê÷râbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPˆ•†÷mãbh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JP÷ƒx läbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JP. ¢–måbh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JP'–fmæbh EDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated"trainer/global_step2:JPF¶Ìnçbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JP›vSÝgèbb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JP€&smébh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JPè¤)änêbi FDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JP?ð Jhëbc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JP˜iìbd ADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated"trainer/global_step2:JP”›÷;oíbj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JP4kkîbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JPÙ  $bïb] :DeviceStatsMonitor.on_train_batch_end/allocation.all.freed"trainer/global_step2:JP ®]žjðbe BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JP.|¥hñbc @DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current"trainer/global_step2:JPþrÍÙjòbe BDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current"trainer/global_step2:JP­@Gnóbi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JPˆH%gôbb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JP »à dõb_ DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPÝÄTÞi€bd ADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed"trainer/global_step2:JPË^gçlbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JP},hf‚ba >DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JP㿬Àqƒbl IDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JP Bk¤d„b_ DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JPß–œm˜bh EDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current"trainer/global_step2:JPW¨y†s™bn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP‹Ü\ ršbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JP~ÿ•g›bb ?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current"trainer/global_step2:JPÔ¿ÍÛjœbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPÜ. hbc @DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak"trainer/global_step2:JPiÇ,pžbk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JPoµû aŸb\ 9DeviceStatsMonitor.on_train_batch_end/segment.all.current"trainer/global_step2:JP´Óüšh bc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JP=îðo¡bj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JP»85.¢) _wandbruntime‚55  _runtime‚55Éâ Y££M YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1186 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 4109284364674 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚917749 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 4186412176384 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 56842360320 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2323780 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 $ _timestamp‚1.788495998025521e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1259954 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚622363 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 trainer/global_step‚49 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1284826 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 56842360320 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 253475840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1284826 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 4186430179328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 4186412176384 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2298906 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚918362 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚613 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 56627420082 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 4243254536704 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1259954 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1149 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚622365 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 4051748976928 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1038952 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 57751578112 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1038954 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 4622501364736 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 57535387746 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚611 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4683068120064 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 _runtime‚ 55.803315058 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 57751578112 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1038952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 229535744 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚295386 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 4244181757440 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4682814644224 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 4108359357650 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2323780 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 4186430179328 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚295997 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 60542815232 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 60313279488 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2298906 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1038954 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736µ4­&¤§M C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1259954 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1149 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚622365 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 4051748976928 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1038952 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 57751578112 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1038954 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 4622501364736 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 57535387746 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚611 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4683068120064 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 _runtime‚ 55.803315058 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 57751578112 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1038952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 229535744 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚295386 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 4244181757440 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4682814644224 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 4108359357650 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2323780 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 4186430179328 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚295997 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 60542815232 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 60313279488 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2298906 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1038954 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1186 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 4109284364674 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚917749 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 4186412176384 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 56842360320 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2323780 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 $ _timestamp‚1.788495998025521e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1259954 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚622363 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 trainer/global_step‚49 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1284826 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 56842360320 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 253475840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1284826 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 4186430179328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 4186412176384 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2298906 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚918362 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚613 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 56627420082 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 4243254536704 FË A¨b< loss/FBt4/assignment_loss"trainer/global_step2:JP¾Õ@©b; loss/FRt1/detection_loss"trainer/global_step2:JP´7_ï7ªb2 loss/total_loss"trainer/global_step2:JP:Ã4C«b> loss/SRqqt3/assignment_loss"trainer/global_step2:JP_ÎêˆA¬b< loss/FBt1/assignment_loss"trainer/global_step2:JP±Jœ@­b; loss/FBt3/detection_loss"trainer/global_step2:JPW8ížA®b< loss/FRt4/assignment_loss"trainer/global_step2:JP _e2@¯b; loss/FRt4/detection_loss"trainer/global_step2:JP7:s@°b; loss/FBt1/detection_loss"trainer/global_step2:JP~ÁêC±b> loss/SRqqt4/assignment_loss"trainer/global_step2:JPeЇ'B²b= loss/SRqqt3/detection_loss"trainer/global_step2:JP†|V×C³b> loss/SRqqt2/assignment_loss"trainer/global_step2:JP>o(ãA´b< loss/FBt2/assignment_loss"trainer/global_step2:JPZ[,èAµb< loss/FBt3/assignment_loss"trainer/global_step2:JP9@¶b; loss/FRt2/detection_loss"trainer/global_step2:JPÖ|fË@·b; loss/FRt3/detection_loss"trainer/global_step2:JP|y7Ä-¸b( epoch"trainer/global_step2:JPÑhy1C¹b> loss/SRqqt1/assignment_loss"trainer/global_step2:JP´ 2>Bºb= loss/SRqqt1/detection_loss"trainer/global_step2:JPÎÁsaB»b= loss/SRqqt4/detection_loss"trainer/global_step2:JP…ÉÝkA¼b< loss/FRt2/assignment_loss"trainer/global_step2:JPðÍÞ@B½b= loss/SRqqt2/detection_loss"trainer/global_step2:JPÁdqé@¾b; loss/FBt2/detection_loss"trainer/global_step2:JPMÊOP.¿) _wandbruntime‚55  _runtime‚55,méøHÀ 1loss/SRqqt1/assignment_loss‚5.437018232142671 0loss/SRqqt1/detection_loss‚3.773781918464823 0loss/SRqqt4/detection_loss‚5.022516322486541 .loss/FRt2/assignment_loss‚7.47527004588734 1loss/SRqqt2/detection_loss‚1.8750555931021355 /loss/FBt2/detection_loss‚11.338809647162755 /loss/FBt4/detection_loss‚2.8406455757248463 /loss/FRt1/assignment_loss‚7.074944412007051 /loss/FRt3/assignment_loss‚7.175544998862526 0loss/FBt4/assignment_loss‚2.6520695199771804 .loss/FRt1/detection_loss‚7.031663318063291 &loss/total_loss‚123.82696924645538 trainer/global_step‚49 _runtime‚ 55.805447249 1loss/SRqqt3/assignment_loss‚5.369011002202188 -loss/FBt1/assignment_loss‚2.7010526006872 .loss/FBt3/detection_loss‚8.145855202561332 % _timestamp‚1.7884959980280817e+09 0loss/FRt4/assignment_loss‚7.3667241681006645 /loss/FRt4/detection_loss‚5.4303371377529635 .loss/FBt1/detection_loss‚8.375231904062359 0loss/SRqqt4/assignment_loss‚5.44747311928693 0loss/SRqqt3/detection_loss‚2.559558133925161 _step‚3 1loss/SRqqt2/assignment_loss‚5.533712069193522 0loss/FBt2/assignment_loss‚2.6754204114278157 0loss/FBt3/assignment_loss‚2.7420923596336726 /loss/FRt2/detection_loss‚3.0203942571174016 /loss/FRt3/detection_loss‚2.7627872966229914 epoch‚0–؈}LÁÆ /loss/FRt1/assignment_loss‚7.074944412007051 /loss/FRt3/assignment_loss‚7.175544998862526 0loss/FBt4/assignment_loss‚2.6520695199771804 .loss/FRt1/detection_loss‚7.031663318063291 &loss/total_loss‚123.82696924645538 trainer/global_step‚49 _runtime‚ 55.805447249 1loss/SRqqt3/assignment_loss‚5.369011002202188 -loss/FBt1/assignment_loss‚2.7010526006872 .loss/FBt3/detection_loss‚8.145855202561332 % _timestamp‚1.7884959980280817e+09 0loss/FRt4/assignment_loss‚7.3667241681006645 /loss/FRt4/detection_loss‚5.4303371377529635 .loss/FBt1/detection_loss‚8.375231904062359 0loss/SRqqt4/assignment_loss‚5.44747311928693 0loss/SRqqt3/detection_loss‚2.559558133925161 _step‚3 1loss/SRqqt2/assignment_loss‚5.533712069193522 0loss/FBt2/assignment_loss‚2.6754204114278157 0loss/FBt3/assignment_loss‚2.7420923596336726 /loss/FRt2/detection_loss‚3.0203942571174016 /loss/FRt3/detection_loss‚2.7627872966229914 epoch‚0 1loss/SRqqt1/assignment_loss‚5.437018232142671 0loss/SRqqt1/detection_loss‚3.773781918464823 0loss/SRqqt4/detection_loss‚5.022516322486541 .loss/FRt2/assignment_loss‚7.47527004588734 1loss/SRqqt2/detection_loss‚1.8750555931021355 /loss/FBt2/detection_loss‚11.338809647162755 /loss/FBt4/detection_loss‚2.8406455757248463ýßûGë‚(:à ‚‘éÔý„Öm disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda1.in‚0 disk.sdb1.out‚0 disk.md0p1.out‚6.375$ memory_percent‚98.63040447235107* proc.memory.percent‚17.732596397399902 proc.cpu.threads‚17 disk.nvme1n1.in‚ 0.0546875 disk.rbd0.out‚0 network.sent‚90133 disk.sda1.out‚0 network.recv‚105960) disk./.usagePercent‚83.70303747436733 disk.nvme0n1.in‚0.03125 disk.nvme0n1.out‚ 73.0546875 disk.md0.out‚6.375( proc.memory.availableMB‚ 224.39453125 cpu‚24.777339310830467 disk.rbd0.in‚0% disk./.usageGB‚2993.6260566711426 disk.sda.in‚0 disk.md0p1.in‚0 disk.sdb.out‚ 6.603515625# proc.memory.rssMB‚ 2905.30859375 disk.sdb2.out‚ 6.603515625 disk.md0.in‚0 disk.dm-0.out‚423.5 disk.sdb.in‚0 disk.sda2.out‚ 6.603515625 disk.nvme1n1.out‚ 350.4453125 disk.rbd2.out‚0 disk.sdb2.in‚0 disk.sda.out‚ 6.603515625 disk.sdb1.in‚0 disk.dm-0.in‚ 0.0859375€-§¼‘Â(:† ‚‘éÔý¸‡o gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.916' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.229 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ègµg‘Ä‚(:† ‘‘éÔ¨äÃk gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.168' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Å‚(:õ ‘‘éÔæôýl disk.sdb1.out‚0 disk.md0p1.out‚ 8.1484375 disk.sdb.out‚ 8.4443359375 disk.sda2.in‚0 disk.nvme0n1.out‚ 145.1484375 disk.nvme1n1.in‚ 0.0546875! disk.nvme1n1.out‚ 440.75390625 disk.rbd0.in‚0 disk.sda2.out‚ 8.4443359375 disk.dm-0.in‚ 0.20703125( proc.memory.availableMB‚ 223.37109375) disk./.usagePercent‚83.70314968124714$ disk./.usageGB‚2993.630069732666 disk.sdb.in‚0 disk.sda.out‚ 8.4443359375 proc.memory.rssMB‚ 2905.3125* proc.memory.percent‚17.732620239257812 proc.cpu.threads‚17 network.sent‚141896 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚24.813472483135442 disk.md0p1.in‚0 disk.rbd2.out‚0 disk.sdb2.in‚0 disk.sdb2.out‚ 8.4443359375 disk.sda.in‚0 disk.sdb1.in‚0$ memory_percent‚98.63665103912354 disk.md0.in‚0 disk.md0.out‚ 8.1484375 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.dm-0.out‚ 585.90234375 network.recv‚113943 disk.nvme0n1.in‚ 0.15234375‚ï'*“Æ‚(:ˆ  ‘éÔßù½k gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚39.0 gpu.0.powerWatts‚178.297' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.57425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z7¨3ìÇ‚(:á  ‘éÔÿÝÂl disk.sda.in‚0 disk.sdb1.out‚0 disk.dm-0.out‚ 640.015625 proc.memory.rssMB‚ 2905.3125 disk.nvme0n1.in‚ 0.15234375 disk.rbd2.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.md0p1.out‚9.484375) disk./.usagePercent‚83.70314872130234% disk./.usageGB‚2993.6300354003906 disk.sdb.in‚0 disk.sda2.in‚0! disk.nvme1n1.out‚ 490.44140625 disk.md0.out‚9.484375 disk.sdb1.in‚0 disk.md0p1.in‚0 disk.sdb2.out‚9.828125 disk.sda1.in‚0 disk.dm-0.in‚ 0.20703125" memory_percent‚98.636794090271 disk.sda.out‚9.828125 proc.cpu.threads‚17( proc.memory.availableMB‚ 223.34765625 network.sent‚146865 network.recv‚119658 disk.sdb.out‚9.828125 disk.sda1.out‚0 cpu‚24.837941170392742 disk.sda2.out‚9.828125 disk.nvme1n1.in‚ 0.0546875 disk.md0.in‚0 disk.sdb2.in‚0* proc.memory.percent‚17.732620239257812! disk.nvme0n1.out‚ 149.57421875 disk.rbd2.in‚0šqü¨‘È‚(:† ¯‘éÔ¬¦Ók gpu.0.gpu‚15.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚39.0 gpu.0.powerWatts‚179.856' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~ìÉ‚(:ý ¯‘éÔí´ül disk.sda.in‚0 disk.dm-0.out‚ 790.6484375 disk.nvme1n1.in‚ 0.0546875 disk.md0.in‚0 disk.rbd2.out‚0% disk./.usageGB‚2993.3508911132812 disk.md0p1.in‚0 disk.dm-0.in‚ 0.20703125 disk.md0.out‚ 10.90234375 disk.sda.out‚11.30517578125 network.recv‚124357 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0$ proc.memory.availableMB‚223.1875 proc.memory.rssMB‚ 2905.3125 network.sent‚153085 disk.md0p1.out‚ 10.90234375 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.sda2.out‚11.30517578125! disk.nvme0n1.out‚ 202.26953125$ memory_percent‚98.63777160644531* proc.memory.percent‚17.732620239257812( disk./.usagePercent‚83.6953437302038 disk.sdb.in‚0 disk.sdb.out‚11.30517578125! disk.nvme1n1.out‚ 588.37890625 disk.sdb2.out‚11.30517578125 cpu‚24.858335769520703 disk.sdb1.in‚0 disk.nvme0n1.in‚ 0.15234375 disk.sdb2.in‚0Ð_½0Ê+ _wandbruntime‚108  _runtime‚108¿ÍäÕÁË» & lr-AdamW/pg2‚8.617288651972115e-05 trainer/global_step‚99 % _timestamp‚1.7884960504745579e+09 _runtime‚ 108.25188812 _step‚4 & lr-AdamW/pg1‚8.617288651972115e-054¦ÅÌ¿ & lr-AdamW/pg1‚8.617288651972115e-05 & lr-AdamW/pg2‚8.617288651972115e-05 trainer/global_step‚99 % _timestamp‚1.7884960504745579e+09 _runtime‚ 108.25188812 _step‚4ªi£Õ0Í+ _wandbruntime‚108  _runtime‚108Æ%à}‚ΧO A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1174 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2026794 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4531665 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2026794 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 8298482914304 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1213548 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235840512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 9043560882688 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8297555706880 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2480025 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 8185765384704 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 117690697216 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9161275520000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 8034981472136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 117454856704 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2504871¤XM±'Ï«O P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 117454856704 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2504871 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1174 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2026794 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4531665 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2026794 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 8298482914304 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1213548 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235840512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 9043560882688 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8297555706880 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2480025 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 8185765384704 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 117690697216 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9161275520000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 8034981472136k¥¾0Ð+ _wandbruntime‚109  _runtime‚109#6啶&Ñ°M KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 8116041346982 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚583683 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2046954 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 247184384 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 118858292736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 223244288 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 118635048448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1808696 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1225615 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚583083 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 113412642854 % _timestamp‚1.7884960516196215e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 8267366467584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚99 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 113839003136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4576682 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚569 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4551808 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 112929785344 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2529728 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2504856 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 8381223473664 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 112929785344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚602 NDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 8380296252928 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4576682 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4551808 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 9133810527232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1211 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1809298 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1225613 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2046952 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚600 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9252421635584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9252668819968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 112504675190 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1174 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2΂"Ò´M UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1225615 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚583083 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 113412642854 % _timestamp‚1.7884960516196215e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 8267366467584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚99 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 113839003136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4576682 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚569 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4551808 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 112929785344 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2529728 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2504856 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 8381223473664 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 112929785344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚602 NDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 8380296252928 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4576682 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4551808 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 9133810527232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1211 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1809298 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1225613 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2046952 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚600 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9252421635584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9252668819968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 112504675190 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1174 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 8116041346982 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚583683 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2046954 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 247184384 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_byt³ÚÓ?Ÿes.small_pool.allocated‚ 118858292736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 223244288 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 118635048448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1808696Ä.0Ó+ _wandbruntime‚109  _runtime‚109:¹‘KÔÅ /loss/FRt1/assignment_loss‚7.224790323348272 0loss/FBt4/assignment_loss‚0.6102558132261038 .loss/FRt1/detection_loss‚5.701591592459452 .loss/FRt4/detection_loss‚3.626944615141205 /loss/FRt3/assignment_loss‚7.071082264184952 0loss/FBt2/assignment_loss‚0.5119925814752395 %loss/total_loss‚129.9344178028283 _step‚7 1loss/SRqqt2/detection_loss‚1.8705532679272754 trainer/global_step‚99 1loss/SRqqt1/assignment_loss‚5.260564698113336 .loss/FRt2/detection_loss‚3.768168514635828 epoch‚0 /loss/FRt2/assignment_loss‚7.417713758680556 1loss/SRqqt2/assignment_loss‚5.389183987741885 1loss/SRqqt4/assignment_loss‚5.415783144178844 0loss/SRqqt1/detection_loss‚3.890743769539727 /loss/FBt1/detection_loss‚26.347829622881754 .loss/FBt2/detection_loss‚12.90705755352974 /loss/FBt4/detection_loss‚2.0715853851947434 /loss/FRt4/assignment_loss‚7.343625006468399 1loss/SRqqt3/assignment_loss‚5.133225930703653 0loss/FBt3/assignment_loss‚0.5453446380794048 1loss/SRqqt4/detection_loss‚3.8765151859039353 .loss/FBt3/detection_loss‚8.318763771653176 % _timestamp‚1.7884960516213937e+09 0loss/FBt1/assignment_loss‚0.7907492135252271 .loss/FRt3/detection_loss‚3.079052892707599 1loss/SRqqt3/detection_loss‚1.7613002715279926 _runtime‚ 109.398384484/µ·)OÕÉ .loss/FBt2/detection_loss‚12.90705755352974 /loss/FBt4/detection_loss‚2.0715853851947434 /loss/FRt4/assignment_loss‚7.343625006468399 1loss/SRqqt3/assignment_loss‚5.133225930703653 0loss/FBt3/assignment_loss‚0.5453446380794048 1loss/SRqqt4/detection_loss‚3.8765151859039353 .loss/FBt3/detection_loss‚8.318763771653176 % _timestamp‚1.7884960516213937e+09 0loss/FBt1/assignment_loss‚0.7907492135252271 .loss/FRt3/detection_loss‚3.079052892707599 1loss/SRqqt3/detection_loss‚1.7613002715279926 _runtime‚ 109.398384484 /loss/FRt1/assignment_loss‚7.224790323348272 0loss/FBt4/assignment_loss‚0.6102558132261038 .loss/FRt1/detection_loss‚5.701591592459452 .loss/FRt4/detection_loss‚3.626944615141205 /loss/FRt3/assignment_loss‚7.071082264184952 0loss/FBt2/assignment_loss‚0.5119925814752395 %loss/total_loss‚129.9344178028283 _step‚7 1loss/SRqqt2/detection_loss‚1.8705532679272754 trainer/global_step‚99 1loss/SRqqt1/assignment_loss‚5.260564698113336 .loss/FRt2/detection_loss‚3.768168514635828 epoch‚0 /loss/FRt2/assignment_loss‚7.417713758680556 1loss/SRqqt2/assignment_loss‚5.389183987741885 1loss/SRqqt4/assignment_loss‚5.415783144178844 0loss/SRqqt1/detection_loss‚3.890743769539727 /loss/FBt1/detection_loss‚26.347829622881754Ï[ǯ‘Ö‚(:† ¾‘éÔåŠïl gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.676' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.419 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}9®] ׂ(:‚ ¾‘éÔ”å¨m disk.sda.in‚0$ memory_percent‚98.63708019256592# proc.memory.rssMB‚ 2905.31640625% disk./.usageGB‚2993.6009368896484 disk.sda2.in‚0 disk.rbd2.out‚0 disk.rbd0.out‚0 disk.sdb1.in‚0 disk.md0p1.out‚ 12.5546875* proc.memory.percent‚17.732644081115723 proc.cpu.threads‚17 disk.md0.out‚ 12.5546875 network.sent‚159870 disk.sdb.out‚13.01806640625! disk.nvme0n1.out‚ 359.55859375 disk.md0.in‚0 disk.sdb2.out‚13.01806640625 disk.sda.out‚13.01806640625 disk.sda1.out‚0 network.recv‚129647) disk./.usagePercent‚83.70233511476332 disk.nvme1n1.in‚ 0.0546875 disk.sdb2.in‚0( proc.memory.availableMB‚ 223.30078125 cpu‚24.87283381364777 disk.md0p1.in‚0 disk.nvme1n1.out‚ 643.34375 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sdb.in‚0 disk.sda2.out‚13.01806640625 disk.sda1.in‚0 disk.dm-0.out‚ 1002.90234375 disk.nvme0n1.in‚ 0.15234375 disk.rbd2.in‚0 disk.dm-0.in‚ 0.20703125éE¹@‘Ø‚(:† Í‘éÔçþÍk gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚38.0 gpu.0.powerWatts‚182.364' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.591 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O5Î Ù‚(:ÿ Í‘éÔ˜ùÇl disk.rbd0.out‚0 disk.dm-0.out‚ 1161.26953125 disk.sdb.in‚0 disk.rbd2.out‚0 disk.sdb2.in‚0 disk.nvme1n1.out‚ 644.203125 disk.rbd2.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0$ memory_percent‚98.63777160644531* proc.memory.percent‚17.732644081115723 network.sent‚208676% disk./.usageGB‚2993.5432777404785 disk.sda.in‚0 disk.sda1.in‚0 disk.sda.out‚14.99560546875 disk.sdb1.in‚0 disk.sdb.out‚14.99560546875 disk.sda2.out‚14.99560546875# proc.memory.rssMB‚ 2905.31640625( disk./.usagePercent‚83.7007229408203 disk.sdb1.out‚0 disk.md0p1.in‚0 disk.sda2.in‚0 disk.md0.out‚ 14.4609375 network.recv‚134582 disk.md0p1.out‚ 14.4609375! disk.nvme0n1.out‚ 517.06640625 disk.nvme1n1.in‚ 0.0546875 disk.md0.in‚0$ proc.memory.availableMB‚223.1875 cpu‚24.886534338237215 disk.sdb2.out‚14.99560546875 disk.dm-0.in‚ 0.20703125 disk.nvme0n1.in‚ 0.15234375 proc.cpu.threads‚17Ø"wœÚ‚(:‘ Ü‘éÔ¹´õl gpu.0.gpu‚51.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚38.0 gpu.0.powerWatts‚117.81' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.452499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬ÝúñÛ‚(:ƒ Ü‘éÔÕ¸Ýn disk.sda2.in‚0! disk.nvme1n1.out‚ 644.29296875 disk.md0p1.in‚0 disk.dm-0.in‚ 0.20703125 disk.dm-0.out‚ 1352.51171875 disk.rbd2.out‚0 network.sent‚217147 disk.sda2.out‚16.68896484375 disk.nvme0n1.out‚ 708.21875 disk.sdb2.out‚16.68896484375 disk.sdb.in‚0 disk.sdb.out‚16.68896484375) disk./.usagePercent‚83.71711293148878 disk.rbd2.in‚0 disk.rbd0.in‚0$ memory_percent‚98.64237308502197 cpu‚24.89845296872858 proc.cpu.threads‚17$ disk./.usageGB‚2994.129463195801 disk.md0.in‚0 disk.md0.out‚ 16.09765625 network.recv‚139282# proc.memory.rssMB‚ 2905.81640625 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sdb1.out‚0 disk.md0p1.out‚ 16.09765625( proc.memory.availableMB‚ 222.43359375* proc.memory.percent‚17.735695838928223 disk.sdb1.in‚0 disk.nvme0n1.in‚ 0.15234375 disk.nvme1n1.in‚ 0.0546875 disk.sdb2.in‚0 disk.sda.out‚16.68896484375 disk.sda1.out‚0-³£r0Ü+ _wandbruntime‚163  _runtime‚163N4‰»ÅÝ¿ ' lr-AdamW/pg1‚0.00012969454637816618 ' lr-AdamW/pg2‚0.00012969454637816618 trainer/global_step‚149 % _timestamp‚1.7884961060857804e+09 _runtime‚ 163.863105007 _step‚8¾“ÉÞà _step‚8 ' lr-AdamW/pg1‚0.00012969454637816618 ' lr-AdamW/pg2‚0.00012969454637816618 trainer/global_step‚149 % _timestamp‚1.7884961060857804e+09 _runtime‚ 163.863105007‚®(É0ß+ _wandbruntime‚163  _runtime‚163Ýp˜¾'à¸O N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 168804954624 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚12435524630528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 167895750144 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2683490 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 % _timestamp‚1.7884961060869663e+09 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 175948911616 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚12434597423104 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3724927 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6784567 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚12266701672960 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3749773 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6759719 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚12266719675904 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚866081 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _step‚9 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3749773 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1816798 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 227451904 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚12435524630528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚609 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 175721459712 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3724927 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3034794 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚866690 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 251392000 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚12266701672960 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13730818956800 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚12266719675904 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚12042663461260 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 167895750144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚12434597423104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2682879 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13554846105088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 168804954624 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚13730567564800 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚13554870045184 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚12041738454444 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚11874473797424 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1816800 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 118772084584G¦Â'á¼O QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚12434597423104 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3724927 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6784567 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚12266701672960 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3749773 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6759719 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚12266719675904 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚866081 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _step‚9 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3749773 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1816798 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 227451904 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚12435524630528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚609 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 175721459712 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3724927 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3034794 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚866690 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 251392000 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚12266701672960 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13730818956800 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚12266719675904 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚12042663461260 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 167895750144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚12434597423104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2682879 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13554846105088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 168804954624 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚13730567564800 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚13554870045184 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚12041738454444 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚11874473797424 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1816800 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 168804954624 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚12435524630528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 167895750144 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2683490 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 % _timestamp‚1.7884961060869663e+09 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 175948911616 O<Ú”0â+ _wandbruntime‚164  _runtime‚164ÓP˜ ãÃM NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2701255 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3749758 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚12121873322266 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚12348338761728 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚12348320758784 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11953491391968 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3054954 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚13645071809536 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚569 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 169926428160 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 169017210368 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6829584 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2700635 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 177123267584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 176893731840 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚13821965541376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3054952 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚12517337969152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3774630 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚13822219017216 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3749758 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚12122798329290 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 229535744 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 169289897962 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 _runtime‚ 164.940229152 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚872390 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚871772 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚12348320758784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚12348338761728 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚12348320758784 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11953491391968 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3054954 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚13645071809536 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚569 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 169926428160 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 169017210368 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6829584 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2700635 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 177123267584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 176893731840 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚13821965541376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3054952 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚12517337969152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3774630 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚13822219017216 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 5 ÄQ0å+ _wandbruntime‚164  _runtime‚164¶jRæÌ .loss/FBt2/detection_loss‚4.969783581793308 1loss/SRqqt3/detection_loss‚1.5128676738822833 .loss/FBt1/detection_loss‚14.61517128845056 .loss/FBt3/detection_loss‚3.251700240308824 epoch‚0 /loss/FRt4/assignment_loss‚7.422330154562896 1loss/SRqqt1/assignment_loss‚3.726939329394588 2loss/SRqqt4/assignment_loss‚3.6109747400089187 1loss/FBt2/assignment_loss‚0.47773698875971604 %loss/total_loss‚99.34340917354982 % _timestamp‚1.7884961071641643e+09 _runtime‚ 164.941160956 1loss/SRqqt2/assignment_loss‚4.079674124717712 1loss/FBt3/assignment_loss‚0.22747246772253318 /loss/FRt2/detection_loss‚3.7280100564161938 1loss/SRqqt4/detection_loss‚3.3513341959641902 .loss/FBt4/detection_loss‚4.998204514827707 /loss/FRt1/assignment_loss‚7.198548812132615 /loss/FRt2/assignment_loss‚7.524334822760688 /loss/FRt3/assignment_loss‚7.203228822121253 0loss/FBt4/assignment_loss‚0.7977192778554227 0loss/SRqqt1/detection_loss‚3.223190523959972 _step‚11 1loss/SRqqt2/detection_loss‚4.0050365558075995 trainer/global_step‚149 2loss/SRqqt3/assignment_loss‚3.2198546703045183 0loss/FBt1/assignment_loss‚0.5650814238664074 /loss/FRt1/detection_loss‚3.8042671978473663 .loss/FRt3/detection_loss‚2.825025207076508 /loss/FRt4/detection_loss‚3.0049225030080327¥ñ`VçÐ 1loss/SRqqt2/detection_loss‚4.0050365558075995 trainer/global_step‚149 2loss/SRqqt3/assignment_loss‚3.2198546703045183 0loss/FBt1/assignment_loss‚0.5650814238664074 /loss/FRt1/detection_loss‚3.8042671978473663 .loss/FRt3/detection_loss‚2.825025207076508 /loss/FRt4/detection_loss‚3.0049225030080327 .loss/FBt2/detection_loss‚4.969783581793308 1loss/SRqqt3/detection_loss‚1.5128676738822833 .loss/FBt1/detection_loss‚14.61517128845056 .loss/FBt3/detection_loss‚3.251700240308824 epoch‚0 /loss/FRt4/assignment_loss‚7.422330154562896 1loss/SRqqt1/assignment_loss‚3.726939329394588 2loss/SRqqt4/assignment_loss‚3.6109747400089187 1loss/FBt2/assignment_loss‚0.47773698875971604 %loss/total_loss‚99.34340917354982 % _timestamp‚1.7884961071641643e+09 _runtime‚ 164.941160956 1loss/SRqqt2/assignment_loss‚4.079674124717712 1loss/FBt3/assignment_loss‚0.22747246772253318 /loss/FRt2/detection_loss‚3.7280100564161938 1loss/SRqqt4/detection_loss‚3.3513341959641902 .loss/FBt4/detection_loss‚4.998204514827707 /loss/FRt1/assignment_loss‚7.198548812132615 /loss/FRt2/assignment_loss‚7.524334822760688 /loss/FRt3/assignment_loss‚7.203228822121253 0loss/FBt4/assignment_loss‚0.7977192778554227 0loss/SRqqt1/detection_loss‚3.223190523959972 _step‚11 .'K ’è‚(:‡ ë‘éÔÞ•˜l gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚37.0 gpu.0.powerWatts‚118.142' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.5355 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0° a{ýé‚(:ò ë‘éԼȖm( proc.memory.availableMB‚ 221.91796875 disk.sda1.out‚0 disk.sda2.in‚0 disk.nvme1n1.in‚ 0.0546875 disk.sdb.out‚ 18.052734375 disk.sdb2.in‚0 disk.sda.in‚0 disk.md0p1.in‚0 disk.sdb.in‚0" proc.memory.rssMB‚ 2905.8203125* proc.memory.percent‚17.735719680786133 network.sent‚221384 disk.sda1.in‚0 disk.rbd0.in‚0 disk.md0.out‚17.40625$ memory_percent‚98.64552021026611 proc.cpu.threads‚17 disk.sda.out‚ 18.052734375 disk.md0p1.out‚17.40625 disk.dm-0.out‚ 1460.62109375 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.sda2.out‚ 18.052734375 disk.nvme0n1.in‚ 0.15234375 disk.nvme0n1.out‚816.125! disk.nvme1n1.out‚ 644.49609375) disk./.usagePercent‚83.71713469023734 disk.sdb2.out‚ 18.052734375 disk.sdb1.in‚0 disk.dm-0.in‚ 0.20703125 disk.rbd2.out‚0 cpu‚24.90819672144952 disk.sdb1.out‚0 disk.md0.in‚0 network.recv‚143938$ disk./.usageGB‚2994.130241394043S·KË’ê‚(:‡ ú‘éÔš°Àk gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚39.0 gpu.0.powerWatts‚163.422' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.8555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰(›“ë‚(:ú ú‘éÔ½ò²l proc.cpu.threads‚17 network.sent‚271252 disk.sda1.out‚0" disk.nvme0n1.out‚ 1038.49609375* proc.memory.percent‚17.735719680786133 network.recv‚150132$ disk./.usageGB‚2994.255096435547 disk.rbd0.out‚0 disk.md0p1.in‚0 disk.sdb.in‚0 disk.nvme0n1.in‚ 0.15234375 disk.md0.out‚ 18.8984375 disk.rbd0.in‚0 disk.nvme1n1.out‚ 644.65625 disk.nvme1n1.in‚ 0.0546875 disk.rbd2.in‚0( proc.memory.availableMB‚ 221.98828125" proc.memory.rssMB‚ 2905.8203125 disk.sda2.out‚ 19.615234375 cpu‚24.916316639620845 disk.sda1.in‚0 disk.dm-0.in‚ 0.20703125) disk./.usagePercent‚83.72062568945367 disk.sda.in‚0 disk.sdb1.out‚0 disk.sdb.out‚ 19.615234375 disk.sda2.in‚0 disk.rbd2.out‚0 disk.sdb2.out‚ 19.615234375 disk.sda.out‚ 19.615234375 disk.sdb1.in‚0 disk.md0p1.out‚ 18.8984375 disk.dm-0.out‚ 1683.15234375 disk.md0.in‚0 disk.sdb2.in‚0$ memory_percent‚98.64509105682373W†Ï“ì‚(:ˆ ‰’éÔõÔÈk gpu.0.gpu‚20.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚39.0 gpu.0.powerWatts‚175.235' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.80875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'YŇ í‚(:‚ ‰’éÔÌÍ“m disk.sdb1.in‚0( proc.memory.availableMB‚ 223.42578125* proc.memory.percent‚17.735719680786133 proc.cpu.threads‚17) disk./.usagePercent‚83.72412319496249 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.sda2.in‚0 disk.nvme1n1.out‚ 645.15625 disk.md0.out‚ 20.66796875 disk.sdb2.in‚0" proc.memory.rssMB‚ 2905.8203125 disk.sdb.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.md0p1.in‚0 cpu‚24.92410842226962 disk.md0p1.out‚ 20.66796875 disk.dm-0.in‚ 0.20703125$ disk./.usageGB‚2994.380184173584 disk.sdb2.out‚21.44287109375 disk.md0.in‚0 disk.sda.in‚0 disk.dm-0.out‚ 1877.66796875 disk.sdb.out‚21.44287109375 network.sent‚276968" disk.nvme0n1.out‚ 1232.51171875 disk.rbd2.out‚0 disk.sda.out‚21.44287109375 disk.sda2.out‚21.44287109375# memory_percent‚98.6363172531128 network.recv‚154432 disk.nvme0n1.in‚ 0.15234375 disk.nvme1n1.in‚ 0.0546875 disk.rbd0.out‚0Èì “î‚(:ˆ ˜’éÔ¢§Æk gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚37.0 gpu.0.powerWatts‚123.435' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.85875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t<& ï‚(:† ˜’éÔÇÊÁl disk.md0.out‚ 22.0703125 disk.sda1.out‚0 network.sent‚283326" disk.nvme0n1.out‚ 1463.63671875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.out‚ 2108.87890625) disk./.usagePercent‚83.73114124465502% disk./.usageGB‚2994.6311836242676 disk.md0.in‚0 disk.sda.out‚22.89794921875 disk.rbd0.in‚0 disk.md0p1.out‚ 22.0703125 disk.dm-0.in‚ 0.20703125 disk.sda2.out‚22.89794921875 disk.rbd2.in‚0 disk.sdb2.in‚0 disk.md0p1.in‚0 disk.sdb.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 0.15234375 disk.nvme1n1.out‚ 645.2421875 disk.sda1.in‚0( proc.memory.availableMB‚ 223.30078125 cpu‚24.931208642074576 disk.sdb2.out‚22.89794921875# proc.memory.rssMB‚ 2905.82421875 network.recv‚159321 disk.nvme1n1.in‚ 0.0546875 disk.rbd2.out‚0 disk.sdb.out‚22.89794921875$ memory_percent‚98.63708019256592* proc.memory.percent‚17.735743522644043 disk.sda.in‚0 disk.rbd0.out‚02Ó/Y0ð+ _wandbruntime‚218  _runtime‚218”üÿDÅñ¿ $ _timestamp‚1.788496161060788e+09 _runtime‚ 218.838055292 _step‚12 ' lr-AdamW/pg1‚0.00017321620623661122 ' lr-AdamW/pg2‚0.00017321620623661122 trainer/global_step‚199¥ÌôHÉòà _step‚12 ' lr-AdamW/pg1‚0.00017321620623661122 ' lr-AdamW/pg2‚0.00017321620623661122 trainer/global_step‚199 $ _timestamp‚1.788496161060788e+09 _runtime‚ 218.838055292 Τâ0ó+ _wandbruntime‚218  _runtime‚218§hÖùTô»O TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚16048495436752 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3576519 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚13 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚16347655964160 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4994675 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚18300383491072 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4042794 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18066155267584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 224892379648 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1157097 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚15825370563984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2420050 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18066131327488 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚16347673967104 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1156471 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚15825353524624 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 224892379648 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4042792 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚16049420443568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚16571639139328 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1179 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9037469 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3577147 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 223983175168 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 223983175168 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4042792 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚16572566346752 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _runtime‚ 218.839255788 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚16571639139328 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 234252163584ÖØ:_Å'õ¿O D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 223983175168 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 223983175168 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4042792 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚16572566346752 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _runtime‚ 218.839255788 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚16571639139328 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 234252163584 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚16048495436752 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3576519 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚13 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚16347655964160 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4994675 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚18300383491072 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4042794 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18066155267584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 224892379648 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1157097 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚15825370563984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2420050 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18066131327488 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚16347673967104 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1156471 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚15825353524624 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 224892379648 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4042792 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚16049420443568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚16571639139328 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1179 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9037469 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3577147 ã^­U0ö+ _wandbruntime‚219  _runtime‚219,2mTË&÷ÅM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚625 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4062952 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1162826 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18156357031936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚569 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚16655306906112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4994660 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3594314 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4062952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237924352 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9057612 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1179 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9082486 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 trainer/global_step‚199 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5019532 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 226013853184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4994660 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2432113 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 235645621760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 % _timestamp‚1.7884961621834514e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16429275049984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5019532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261864448 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16129555311598 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 235407697408 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 224259185406 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚16429275049984 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 226013853184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15904371119168 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2432115 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4994660 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3594314 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4062952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237924352 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9057612 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1179 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9082486 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 trainer/global_step‚199 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5019532 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 226013853184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4994660 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2432113 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 235645621760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 % _timestamp‚1.7884961621834514e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16429275049984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5019532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261864448 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16129555311598 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 235407697408 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 224259185406 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚16429275049984 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 226013853184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15904371119168 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2432115 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 E?DeviceStatsMonitor.on_train_A+Fžbatch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚569 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚166553069061123?™m0ù+ _wandbruntime‚219  _runtime‚219r·!êPúÊ /loss/FRt1/assignment_loss‚7.178810882568359 2loss/SRqqt1/assignment_loss‚3.4576613643895024 0loss/FBt3/assignment_loss‚0.3605380333072664 /loss/FBt3/detection_loss‚1.8598720245667346 epoch‚0 1loss/SRqqt1/detection_loss‚3.7236521632775017 0loss/SRqqt4/detection_loss‚3.412742587737739 1loss/SRqqt3/assignment_loss‚3.286028029213489 .loss/FRt1/detection_loss‚5.043165810902914 .loss/FRt2/assignment_loss‚7.35771730211046 1loss/SRqqt2/assignment_loss‚3.945941617091497 0loss/FBt1/assignment_loss‚0.6418256720438983 .loss/FRt3/detection_loss‚2.737024623087861 trainer/global_step‚199 /loss/FRt3/assignment_loss‚7.174014550989324 0loss/FBt2/assignment_loss‚0.7796819459929664 1loss/SRqqt3/detection_loss‚1.9469999450613076 .loss/FBt2/detection_loss‚11.22542276765619 .loss/FBt4/detection_loss‚15.57954752445221 &loss/total_loss‚108.82501353525805 % _timestamp‚1.7884961621866686e+09 2loss/SRqqt4/assignment_loss‚3.3452356408039727 .loss/FRt2/detection_loss‚2.934151867887488 /loss/FRt4/detection_loss‚2.7259359518855306 1loss/SRqqt2/detection_loss‚1.6959381971983276 /loss/FBt1/detection_loss‚10.155271372374367 _runtime‚ 219.963784592 _step‚15 /loss/FRt4/assignment_loss‚7.296966374930689 0loss/FBt4/assignment_loss‚0.9608672857284546DÜÝMTûÎ .loss/FRt2/detection_loss‚2.934151867887488 /loss/FRt4/detection_loss‚2.7259359518855306 1loss/SRqqt2/detection_loss‚1.6959381971983276 /loss/FBt1/detection_loss‚10.155271372374367 _runtime‚ 219.963784592 _step‚15 /loss/FRt4/assignment_loss‚7.296966374930689 0loss/FBt4/assignment_loss‚0.9608672857284546 /loss/FRt1/assignment_loss‚7.178810882568359 2loss/SRqqt1/assignment_loss‚3.4576613643895024 0loss/FBt3/assignment_loss‚0.3605380333072664 /loss/FBt3/detection_loss‚1.8598720245667346 epoch‚0 1loss/SRqqt1/detection_loss‚3.7236521632775017 0loss/SRqqt4/detection_loss‚3.412742587737739 1loss/SRqqt3/assignment_loss‚3.286028029213489 .loss/FRt1/detection_loss‚5.043165810902914 .loss/FRt2/assignment_loss‚7.35771730211046 1loss/SRqqt2/assignment_loss‚3.945941617091497 0loss/FBt1/assignment_loss‚0.6418256720438983 .loss/FRt3/detection_loss‚2.737024623087861 trainer/global_step‚199 /loss/FRt3/assignment_loss‚7.174014550989324 0loss/FBt2/assignment_loss‚0.7796819459929664 1loss/SRqqt3/detection_loss‚1.9469999450613076 .loss/FBt2/detection_loss‚11.22542276765619 .loss/FBt4/detection_loss‚15.57954752445221 &loss/total_loss‚108.82501353525805 % _timestamp‚1.7884961621866686e+09 2loss/SRqqt4/assignment_loss‚3.3452356408039727 ¶¦ “ü‚(:ˆ §’éÔ¶ù·m gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚37.0 gpu.0.powerWatts‚119.091' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.77275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿ôë-ý‚(:ú §’éÔÐî·n disk.md0p1.out‚23.625 network.recv‚164141 disk.sdb1.out‚0 disk.md0p1.in‚0 disk.nvme1n1.out‚ 645.8671875 disk.md0.out‚23.625 disk.rbd2.out‚0 disk.sdb2.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.735743522644043 disk.sda2.out‚24.50634765625 disk.dm-0.in‚ 0.20703125 disk.dm-0.out‚ 2307.453125 network.sent‚302772 disk.sdb2.out‚24.50634765625% disk./.usageGB‚2994.6301651000977 disk.sdb.out‚24.50634765625 disk.sda2.in‚0 disk.sda.in‚0 disk.sda.out‚24.50634765625$ memory_percent‚98.63829612731934' proc.memory.availableMB‚ 223.1015625# proc.memory.rssMB‚ 2905.82421875) disk./.usagePercent‚83.73111276629295 disk.nvme1n1.in‚ 0.0546875 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sdb1.in‚0 cpu‚24.935928356612745! disk.nvme0n1.out‚ 1661.5859375 disk.sda1.in‚0 disk.sdb.in‚0 disk.nvme0n1.in‚ 0.15234375 disk.md0.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17,[„Vþ‚(:ú ¶’éÔ¶›£m disk.rbd0.in‚0 disk.md0p1.in‚0 disk.md0.out‚ 25.2578125' proc.memory.availableMB‚ 223.0703125 disk.nvme1n1.out‚ 645.9765625 disk.sda1.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 0.20703125* proc.memory.percent‚17.735743522644043 proc.cpu.threads‚17 network.sent‚339628 disk.sda1.out‚0 disk.rbd0.out‚0 disk.md0p1.out‚ 25.2578125 cpu‚24.940722746807676 network.recv‚170291 disk.sdb2.in‚0 disk.sda2.out‚ 26.193359375% disk./.usageGB‚2994.6301651000977 disk.nvme0n1.in‚ 0.15234375 disk.sdb2.out‚ 26.193359375 disk.dm-0.out‚ 2534.7109375 disk.sdb.in‚0 disk.sdb.out‚ 26.193359375 disk.md0.in‚0 disk.rbd2.out‚0 disk.sdb1.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2905.82421875 disk.nvme1n1.in‚ 0.0546875 disk.rbd2.in‚0 disk.sda.in‚0$ memory_percent‚98.63848686218262) disk./.usagePercent‚83.73111276629295 disk.nvme0n1.out‚ 1888.734375 disk.sda.out‚ 26.193359375Ñ`ϧ’ÿ‚(:‡ ¶’éÔ»«ªm gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.434' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.8585 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö[á퀂(:ö Å’éÔ¾ìül disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0.in‚0$ proc.memory.availableMB‚223.0625 disk.sda.in‚0 disk.rbd2.in‚0 network.recv‚174242 disk.nvme1n1.out‚ 646.109375 disk.sdb2.in‚0$ memory_percent‚98.63853454589844 cpu‚24.944841162956696 network.sent‚347496 disk.sda1.in‚0 disk.md0p1.in‚0 disk.dm-0.out‚ 2700.109375 disk.nvme0n1.out‚2054 disk.sdb2.out‚27.98486328125* proc.memory.percent‚17.735767364501953 proc.cpu.threads‚17 disk.sda1.out‚0 disk.dm-0.in‚ 61.18359375 disk.sdb.in‚0 disk.nvme1n1.in‚ 5.83203125 disk.rbd2.out‚0) disk./.usagePercent‚83.69922756015995 disk.sda.out‚27.98486328125 disk.rbd0.out‚0 disk.md0p1.out‚ 26.9765625 disk.nvme0n1.in‚ 55.3515625! proc.memory.rssMB‚ 2905.828125% disk./.usageGB‚2993.4897956848145 disk.sdb1.in‚0 disk.sdb.out‚27.98486328125 disk.sda2.out‚27.98486328125 disk.md0.out‚ 26.9765625 disk.sda2.in‚0ü„Zב‚(:† Å’éÔЙ³m gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.916' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.229 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$'}Ô‚‚(:… Ô’éÔšˆÊl$ memory_percent‚98.63860607147217 disk.sda.in‚0 disk.sda.out‚ 29.9267578125 disk.rbd0.out‚0" disk.nvme0n1.out‚ 2272.39453125 disk.rbd2.in‚0( proc.memory.availableMB‚ 223.05078125 disk.sda1.out‚0 disk.sdb.in‚0 disk.rbd2.out‚0 network.sent‚351733 network.recv‚179281% disk./.usageGB‚2993.5760498046875 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sdb1.out‚0 cpu‚24.94855957477294 disk.dm-0.in‚ 61.18359375! disk.nvme1n1.out‚ 646.19140625 disk.md0.out‚ 28.84765625 proc.cpu.threads‚17) disk./.usagePercent‚83.70163926145176 disk.sdb1.in‚0 disk.nvme1n1.in‚ 5.83203125 disk.sdb2.in‚0 disk.sdb2.out‚ 29.9267578125# proc.memory.rssMB‚ 2905.83203125 disk.md0p1.out‚ 28.84765625 disk.sdb.out‚ 29.9267578125 disk.sda2.in‚0 disk.sda2.out‚ 29.9267578125* proc.memory.percent‚17.735791206359863 disk.md0p1.in‚0 disk.dm-0.out‚ 2918.5859375 disk.nvme0n1.in‚ 55.3515625 disk.md0.in‚0lF4‰’ƒ‚(:‡ Ô’éÔ»¸Êm gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚39.0 gpu.0.powerWatts‚174.562' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.6405 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z„0„+ _wandbruntime‚274  _runtime‚274³ømÆ…À _runtime‚ 274.079739154 _step‚16 ' lr-AdamW/pg1‚0.00021673786609505624 ' lr-AdamW/pg2‚0.00021673786609505624 trainer/global_step‚249 % _timestamp‚1.7884962163024127e+09쇵Ã栀 _step‚16 ' lr-AdamW/pg1‚0.00021673786609505624 ' lr-AdamW/pg2‚0.00021673786609505624 trainer/global_step‚249 % _timestamp‚1.7884962163024127e+09 _runtime‚ 274.079739154¯G?0‡+ _wandbruntime‚274  _runtime‚274Ê—2ãÄ'ˆ¾O TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚20708680855552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚20055252419060 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 279927134692 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5050792 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 280070600192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 280070600192 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6214731 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6214731 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚20428628258304 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚19776233251824 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5050794 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 292808438272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 280979804672 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11290371 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6239577 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚20428610255360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1447711 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚22870478477312 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _step‚17 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 229549056 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6239577 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3023300 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 $ _timestamp‚1.788496216303191e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 253489152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚20428628258304 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚645 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚20709608062976 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20708680855552 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1447066 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3023298 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 279019167236'ý½ïÈ'‰ÂO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 229549056 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6239577 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3023300 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 $ _timestamp‚1.788496216303191e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 253489152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚20428628258304 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚645 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚20709608062976 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20708680855552 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1447066 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3023298 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 279019167236 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚20708680855552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚20055252419060 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 279927134692 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5050792 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 280070600192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 280070600192 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6214731 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6214731 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚20428628258304 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚19776233251824 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5050794 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 292808438272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 280979804672 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11290371 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6239577 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚20428610255360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1447711 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚22870478477312 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _step‚17{€%u0Š+ _wandbruntime‚275  _runtime‚275Å›½?r ‹ÈM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 282101278208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6264434 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242990080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚22667666194432 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚22667642254336 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚22961885670912 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚20510247344128 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4488321 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11310514 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11335388 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3035365 OGDeviceStatsMonitor.on_trפí‡\ain_batch_end/inactive_split.small_pool.current‚628 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5070952 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6239562 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 294219476480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 $ _timestamp‚1.788496217296497e+09 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚20791421401600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 280136440514 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3035363 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚20510229341184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚20135387286882 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20792348622336 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 281044408178 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚20792348622336 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11310514 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚22961642680832 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚20510247344128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚20136312293906 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚19855267885728 _runtime‚ 275.074394526 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5070954 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 219049984 trainer/global_step‚249 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11335388 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚19855250846368 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5070954 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚569 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 281192060416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 281192060416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5070952 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 294000426496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4488321 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11310514 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11335388 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3035365 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5070952 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6239562 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 294219476480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 $ _timestamp‚1.788496217296497e+09 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚20791421401600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 280136440514 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3035363 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚20510229341184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚20135387286882 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20792348622336 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 281044408178 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚20792348622336 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11310514 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚22961642680832 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚20510247344128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚20136312293906 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚19855267885728 _runtime‚ 275.074394526 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5070954 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 219049984 trainer/global_step‚249 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11335388 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚19855250846368 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5070954 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚569 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 281192060416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 281192060416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0¶;°0+ _wandbruntime‚275  _runtime‚275åÈ… MŽÇ _runtime‚ 275.075542461 2loss/SRqqt4/assignment_loss‚3.6001622729831273 /loss/FRt2/assignment_loss‚7.436658436601812 epoch‚0 /loss/FRt1/detection_loss‚3.5178996920585632 .loss/FBt2/detection_loss‚4.943170703450838 % _timestamp‚1.7884962172985027e+09 1loss/FBt3/assignment_loss‚0.47742871087393723 1loss/SRqqt4/detection_loss‚3.5656369010607403 .loss/FRt1/assignment_loss‚7.18612630550678 /loss/FRt3/assignment_loss‚7.154492362600858 0loss/FRt4/assignment_loss‚7.2736444570580305 1loss/SRqqt1/assignment_loss‚3.462495916029986 0loss/SRqqt1/detection_loss‚3.203173853603064 $loss/total_loss‚89.7347463708675 1loss/SRqqt2/assignment_loss‚3.916567881902059 2loss/SRqqt3/assignment_loss‚3.0661886846515496 0loss/FBt4/assignment_loss‚0.8688001453876495 .loss/FRt3/detection_loss‚2.292727233445058 trainer/global_step‚249 _step‚19 0loss/FBt1/assignment_loss‚0.5136690024402923 0loss/FBt2/assignment_loss‚0.2498000247195402 0loss/SRqqt3/detection_loss‚1.453409587816847 .loss/FBt1/detection_loss‚6.606847800612449 .loss/FRt2/detection_loss‚3.595755113119429 .loss/FRt4/detection_loss‚3.276192176706937 0loss/SRqqt2/detection_loss‚2.774692289038944 /loss/FBt3/detection_loss‚1.5258138021453993 .loss/FBt4/detection_loss‚7.773393017053604¬±>²QË 0loss/FBt1/assignment_loss‚0.5136690024402923 0loss/FBt2/assignment_loss‚0.2498000247195402 0loss/SRqqt3/detection_loss‚1.453409587816847 .loss/FBt1/detection_loss‚6.606847800612449 .loss/FRt2/detection_loss‚3.595755113119429 .loss/FRt4/detection_loss‚3.276192176706937 0loss/SRqqt2/detection_loss‚2.774692289038944 /loss/FBt3/detection_loss‚1.5258138021453993 .loss/FBt4/detection_loss‚7.773393017053604 _runtime‚ 275.075542461 2loss/SRqqt4/assignment_loss‚3.6001622729831273 /loss/FRt2/assignment_loss‚7.436658436601812 epoch‚0 /loss/FRt1/detection_loss‚3.5178996920585632 .loss/FBt2/detection_loss‚4.943170703450838 % _timestamp‚1.7884962172985027e+09 1loss/FBt3/assignment_loss‚0.47742871087393723 1loss/SRqqt4/detection_loss‚3.5656369010607403 .loss/FRt1/assignment_loss‚7.18612630550678 /loss/FRt3/assignment_loss‚7.154492362600858 0loss/FRt4/assignment_loss‚7.2736444570580305 1loss/SRqqt1/assignment_loss‚3.462495916029986 0loss/SRqqt1/detection_loss‚3.203173853603064 $loss/total_loss‚89.7347463708675 1loss/SRqqt2/assignment_loss‚3.916567881902059 2loss/SRqqt3/assignment_loss‚3.0661886846515496 0loss/FBt4/assignment_loss‚0.8688001453876495 .loss/FRt3/detection_loss‚2.292727233445058 trainer/global_step‚249 _step‚19Ûvu=‚(:† ã’éÔÕñœm disk.sdb2.in‚0 disk.md0p1.out‚ 30.3046875* proc.memory.percent‚17.735791206359863 disk.sda2.in‚0 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚83.68682571344803 disk.sdb1.out‚0 proc.cpu.threads‚17 network.sent‚370496 disk.sda1.out‚0 disk.md0p1.in‚0 cpu‚24.95174273347331 disk.sdb.in‚0 disk.sda2.out‚31.45947265625 disk.md0.out‚ 30.3046875 disk.rbd2.out‚0" disk.nvme0n1.out‚ 2337.66015625 disk.md0.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 61.18359375$ memory_percent‚98.63793849945068( proc.memory.availableMB‚ 223.16015625 network.recv‚183155 disk.sdb.out‚31.45947265625 disk.nvme1n1.in‚ 5.83203125 disk.nvme1n1.out‚ 646.3828125 disk.sdb2.out‚31.45947265625 disk.sdb1.in‚0$ disk./.usageGB‚2993.046245574951 disk.nvme0n1.in‚ 55.3515625 disk.sda.out‚31.45947265625 disk.dm-0.out‚ 2984.04296875# proc.memory.rssMB‚ 2905.83203125žÙ‘‘‚(:† ã’éÔÜÀªm gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚39.0 gpu.0.powerWatts‚176.892' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.223 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸB­ç’’‚(:‡ ò’éÔÑ’Æk gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚39.0 gpu.0.powerWatts‚178.914' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.7285 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê&ê· “‚(: ò’éÔß»‡m disk.sda1.in‚0" disk.nvme0n1.out‚ 2393.49609375 disk.sdb2.out‚32.76123046875 network.recv‚189334 disk.sdb1.in‚0 disk.sda2.out‚32.76123046875 disk.nvme0n1.in‚ 55.3515625' proc.memory.availableMB‚ 222.8828125 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.nvme1n1.out‚ 646.53125# memory_percent‚98.6396312713623* proc.memory.percent‚17.735791206359863( disk./.usagePercent‚83.6842817530964 disk.sda.in‚0 disk.sdb1.out‚0# proc.memory.rssMB‚ 2905.83203125 disk.rbd2.out‚0% disk./.usageGB‚2992.9552612304688 disk.rbd0.in‚0 disk.md0p1.in‚0 disk.md0p1.out‚ 31.546875 disk.dm-0.out‚ 3040.02734375 disk.sdb.out‚32.76123046875 disk.sda2.in‚0 disk.sda1.out‚0 disk.nvme1n1.in‚ 5.83203125 disk.md0.in‚0 disk.sdb2.in‚0 cpu‚24.954764535060512 disk.sda.out‚32.76123046875 disk.dm-0.in‚ 61.18359375 disk.sdb.in‚0 disk.md0.out‚ 31.546875 disk.rbd2.in‚0 network.sent‚408762‰.<ˆÿ”‚(:ô “éÔ³˜Îl" disk.nvme0n1.out‚ 2394.32421875% disk./.usageGB‚2993.1412811279297 disk.md0p1.out‚33.6875* proc.memory.percent‚17.735791206359863 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.nvme0n1.in‚ 55.3515625! disk.nvme1n1.out‚ 718.71484375 proc.cpu.threads‚17 network.sent‚414315 disk.sdb.in‚0 disk.nvme1n1.in‚ 5.83203125 disk.md0.in‚0 disk.md0.out‚33.6875 disk.dm-0.in‚ 61.18359375 disk.sda2.out‚ 34.97265625 disk.rbd0.out‚0 disk.sdb1.out‚0 disk.sdb2.out‚ 34.97265625 disk.sda.out‚ 34.97265625 disk.sda1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚83.68948294728477 disk.dm-0.out‚ 3113.0390625 disk.rbd2.in‚0# proc.memory.rssMB‚ 2905.83203125 disk.sdb2.in‚0 disk.sda.in‚0 disk.rbd2.out‚0& proc.memory.availableMB‚ 224.515625 cpu‚24.957561151169937 network.recv‚193051 disk.md0p1.in‚0 disk.sdb.out‚ 34.97265625 disk.sda2.in‚0$ memory_percent‚98.62966537475586È ­„“•‚(:ˆ “éÔžßõl gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚39.0 gpu.0.powerWatts‚177.411' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.35275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`tÀV0–+ _wandbruntime‚328  _runtime‚328ñnjÄ—¾ _runtime‚ 328.779540456 _step‚20 & lr-AdamW/pg1‚0.0002602595259535013 & lr-AdamW/pg2‚0.0002602595259535013 trainer/global_step‚299 % _timestamp‚1.7884962710022159e+09ÊAæŸÈ˜Â % _timestamp‚1.7884962710022159e+09 _runtime‚ 328.779540456 _step‚20 & lr-AdamW/pg1‚0.0002602595259535013 & lr-AdamW/pg2‚0.0002602595259535013 trainer/global_step‚299°ôn0™+ _wandbruntime‚328  _runtime‚328}…¬Öš¿O P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7459633 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5366110 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚23727130018384 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚27440311069184 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1221 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚27440057580032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚299 _step‚21 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13518425 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6058794 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7484479 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1739560 _runtime‚ 328.781112293 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 337067229696 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13543273 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24845722571776 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚24509582549504 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 229549056 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 253489152 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6058792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5365472 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3626548 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚24846649779200 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚24062009401368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 336158025216 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚27088701772288 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_spli„Iït.large_pool.allocated‚3626550 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 335804389800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 334896422344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚24845722571776 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚23727112979024 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24846649779200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚24062934408184 % _timestamp‚1.7884962710036652e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚24509582549504 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 336158025216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0}lP¸É'›ÃO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 336158025216 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚27088701772288 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3626550 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 335804389800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 334896422344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚24845722571776 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚23727112979024 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24846649779200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚24062934408184 % _timestamp‚1.7884962710036652e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚24509582549504 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 336158025216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7459633 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5366110 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚23727130018384 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚27440311069184 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1221 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚27440057580032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚299 _step‚21 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13518425 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6058794 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7484479 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1739560 _runtime‚ 328.781112293 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 337067229696 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13543273 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24845722571776 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚24509582549504 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 229549056 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 253489152 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6058792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5365472 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3626548 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚24846649779200 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚24062009401368Î…ýÙ0œ+ _wandbruntime‚329  _runtime‚329¡hæþÎ&ÈM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5384010 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13588290 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6078952 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 $ _timestamp‚1.788496271971748e+09 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚24928463117824 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6078952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1221 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 249281536 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27178927476736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3638613 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 336921663286 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚299 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 _step‚22 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7484464 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 352525753344 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 338188703232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚24929390338560 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1744763 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7484464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5383376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚569 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 337279485440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 352751094784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 225341440 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 _runtime‚ 329.749870276 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚23806147612928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 336013695622 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚24929390338560 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6078954 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1745395 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚23806130573568 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27178951416832 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7509336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚24142144269190 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13588290 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 338188703232 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3638615 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6078954 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27531702511616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615v`rÒ&žÌM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6078954 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1745395 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚23806130573568 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27178951416832 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7509336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚24142144269190 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13588290 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 338188703232 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3638615 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6078954 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27531702511616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5384010 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13588290 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6078952 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 $ _timestamp‚1.788496271971748e+09 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚24928463117824 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6078952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1221 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 249281536 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27178927476736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3638613 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 336921663286 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚299 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 _step‚22 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7484464 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 352525753344 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 338188703232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚24929390338560 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1744763 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7484464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5383376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚569 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 337279485440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 352751094784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 225341440 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 _runtime‚ 329.749870276 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚23806147612928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 336013695622 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚24929390338560 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2a˜vb0Ÿ+ _wandbruntime‚329  _runtime‚329SqÔ Ì 2loss/SRqqt4/assignment_loss‚3.2735479692133462 /loss/FBt3/assignment_loss‚0.546063298709436 .loss/FBt1/detection_loss‚16.75409142971039 &loss/total_loss‚103.12419133948457 2loss/SRqqt1/assignment_loss‚3.3718559850345957 .loss/FRt2/detection_loss‚4.010047323609653 1loss/SRqqt2/detection_loss‚2.5213173923793986 /loss/FRt2/assignment_loss‚7.389056343781321 /loss/FRt3/assignment_loss‚7.106161107619603 /loss/FRt4/assignment_loss‚7.219822388069303 1loss/SRqqt2/assignment_loss‚3.916186ã…?¦;617266747 1loss/SRqqt3/detection_loss‚1.8674607298462027 .loss/FBt4/detection_loss‚9.848880946637781 /loss/FBt4/assignment_loss‚0.651483303859095 /loss/FBt3/detection_loss‚2.9255378028892762 % _timestamp‚1.7884962719737334e+09 1loss/FBt1/assignment_loss‚0.36295256462071845 1loss/FBt2/assignment_loss‚0.28752307044945796 1loss/SRqqt4/detection_loss‚3.7345384017723364 epoch‚0 trainer/global_step‚299 2loss/SRqqt3/assignment_loss‚3.0461027835732075 /loss/FRt3/detection_loss‚2.8730311593972147 /loss/FRt4/detection_loss‚2.7899309499006644 _runtime‚ 329.75077008 .loss/FRt1/detection_loss‚2.691518948191688 _step‚23 /loss/FRt1/assignment_loss‚7.187973771776472 1loss/SRqqt1/detection_loss‚3.4684641388329593 .loss/FBt2/detection_loss‚5.280642912343696Y®1V¡Ð /loss/FRt4/assignment_loss‚7.219822388069303 1loss/SRqqt2/assignment_loss‚3.916186617266747 1loss/SRqqt3/detection_loss‚1.8674607298462027 .loss/FBt4/detection_loss‚9.848880946637781 /loss/FBt4/assignment_loss‚0.651483303859095 /loss/FBt3/detection_loss‚2.9255378028892762 % _timestamp‚1.7884962719737334e+09 1loss/FBt1/assignment_loss‚0.36295256462071845 1loss/FBt2/assignment_loss‚0.28752307044945796 1loss/SRqqt4/detection_loss‚3.7345384017723364 epoch‚0 trainer/global_step‚299 2loss/SRqqt3/assignment_loss‚3.0461027835732075 /loss/FRt3/detection_loss‚2.8730311593972147 /loss/FRt4/detection_loss‚2.7899309499006644 _runtime‚ 329.75077008 .loss/FRt1/detection_loss‚2.691518948191688 _step‚23 /loss/FRt1/assignment_loss‚7.187973771776472 1loss/SRqqt1/detection_loss‚3.4684641388329593 .loss/FBt2/detection_loss‚5.280642912343696 2loss/SRqqt4/assignment_loss‚3.2735479692133462 /loss/FBt3/assignment_loss‚0.546063298709436 .loss/FBt1/detection_loss‚16.75409142971039 &loss/total_loss‚103.12419133948457 2loss/SRqqt1/assignment_loss‚3.3718559850345957 .loss/FRt2/detection_loss‚4.010047323609653 1loss/SRqqt2/detection_loss‚2.5213173923793986 /loss/FRt2/assignment_loss‚7.389056343781321 /loss/FRt3/assignment_loss‚7.106161107619603­8Æú ¢‚(:þ “éÔïíòl disk.nvme1n1.in‚ 5.83203125 disk.sdb2.in‚0 disk.md0p1.out‚ 34.98046875 disk.rbd2.out‚0 cpu‚24.96071651287079 proc.cpu.threads‚17% disk./.usageGB‚2993.1306190490723 disk.sda1.out‚0 disk.md0p1.in‚0 disk.md0.out‚ 34.98046875* proc.memory.percent‚17.735791206359863 disk.sdb2.out‚36.32373046875 disk.rbd0.out‚0) disk./.usagePercent‚83.68918483109749 disk.sda.out‚36.32373046875 disk.dm-0.out‚ 3271.36328125 disk.sda2.in‚0 disk.nvme0n1.in‚ 55.3515625" disk.nvme0n1.out‚ 2394.36328125 network.sent‚418653 network.recv‚197746 disk.sda1.in‚0 disk.sdb1.out‚0 disk.nvme1n1.out‚877 disk.md0.in‚0 disk.rbd2.in‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb.in‚0 disk.sda2.out‚36.32373046875$ memory_percent‚98.63262176513672% proc.memory.availableMB‚ 224.03125# proc.memory.rssMB‚ 2905.83203125 disk.dm-0.in‚ 61.18359375 disk.sdb.out‚36.32373046875iSÓ“£‚(:ˆ “éÔèÁ¥m gpu.0.gpu‚25.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚39.0 gpu.0.powerWatts‚180.473' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.11825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºtT铤‚(:ˆ Ÿ“éÔøÀæl gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚36.0 gpu.0.powerWatts‚117.525' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.38125 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´C¿Z¥‚(:ƒ Ÿ“éÔ×Íòm disk.rbd0.out‚0 disk.dm-0.out‚ 3311.59765625 disk.nvme1n1.in‚ 5.83203125* proc.memory.percent‚17.735791206359863 disk.dm-0.in‚ 61.18359375 disk.nvme0n1.in‚ 55.3515625 disk.sdb.out‚37.80126953125$ disk./.usageGB‚2993.255683898926 disk.md0.in‚0 disk.rbd2.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sdb1.in‚0! disk.nvme1n1.out‚ 916.91015625# proc.memory.rssMB‚ 2905.83203125 disk.nvme0n1.out‚ 2394.6875 disk.sda1.in‚0 disk.sdb.in‚0 proc.cpu.threads‚17 network.recv‚202863 disk.sda.in‚0 disk.sdb1.out‚0 disk.md0p1.out‚ 36.39453125$ memory_percent‚98.63038063049316 network.sent‚470013 disk.md0.out‚ 36.39453125 disk.sda.out‚37.80126953125 cpu‚24.96284310433668 disk.rbd2.out‚0 disk.sdb2.in‚0 disk.md0p1.in‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 224.3984375 disk.sda2.out‚37.80126953125( disk./.usagePercent‚83.6926816966431 disk.sdb2.out‚37.80126953125xÑA ‘¦‚(:† ®“éÔ¨Üõl gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚37.0 gpu.0.powerWatts‚120.31' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.0775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01ò罧‚(:ö ®“éÔü©m disk.md0.out‚ 37.73828125 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.nvme1n1.in‚ 5.83203125* proc.memory.percent‚17.735791206359863 cpu‚24.96566414815732 disk.sda.in‚0 disk.sdb1.out‚0 disk.nvme0n1.in‚ 55.3515625' proc.memory.availableMB‚ 223.4921875 disk.sda.out‚ 39.20703125 disk.sdb.out‚ 39.20703125 disk.sda2.out‚ 39.20703125 disk.rbd0.in‚0 disk.md0p1.in‚0 disk.sdb.in‚0# proc.memory.rssMB‚ 2905.83203125 network.sent‚474484 network.recv‚207942 disk.sdb2.out‚ 39.20703125 disk.md0.in‚0 proc.cpu.threads‚17 disk.sdb2.in‚0 disk.sdb1.in‚0$ disk./.usageGB‚2993.630416870117 disk.sda1.out‚0 disk.rbd0.out‚0 disk.md0p1.out‚ 37.73828125 disk.dm-0.in‚ 61.18359375$ memory_percent‚98.63591194152832) disk./.usagePercent‚83.70315938735556 disk.dm-0.out‚ 3533.84375 disk.nvme0n1.out‚ 2394.734375 disk.nvme1n1.out‚ 1139.109375U@a‚(:ˆ ½“éÔÒÙ‚m gpu.0.gpu‚32.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.405776977539062* gpu.0.memoryAllocatedBytes‚ 14951448576 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.157' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.78925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+¦e©‚(:ü ½“éÔÕàèm disk.dm-0.out‚ 3684.609375( proc.memory.availableMB‚ 223.86328125 network.sent‚480086 disk.sda.in‚0 disk.rbd0.out‚0 disk.md0p1.in‚0 disk.rbd2.out‚0 disk.sda1.in‚0 disk.md0p1.out‚ 39.4921875 disk.sda2.out‚ 41.0224609375$ memory_percent‚98.63364696502686 disk.md0.in‚0 disk.sdb2.out‚ 41.0224609375 disk.sdb.in‚0# proc.memory.rssMB‚ 2905.83203125$ disk./.usageGB‚2993.881435394287 disk.nvme1n1.in‚ 5.83203125 proc.cpu.threads‚17 disk.nvme0n1.out‚ 2394.921875 disk.md0.out‚ 39.4921875 disk.sda.out‚ 41.0224609375 disk.rbd0.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.735791206359863 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sdb.out‚ 41.0224609375 disk.nvme0n1.in‚ 55.3515625 disk.nvme1n1.out‚ 1289.6875 cpu‚24.96813568678178 disk.sdb2.in‚0 network.recv‚211659) disk./.usagePercent‚83.71017797035077 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 61.18359375Ë0ËÈ0ª+ _wandbruntime‚387  _runtime‚387Ò½A?Å«¿ ' lr-AdamW/pg1‚0.00030378118581194627 ' lr-AdamW/pg2‚0.00030378118581194627 trainer/global_step‚349 % _timestamp‚1.7884963292569473e+09 _runtime‚ 387.03417972 _step‚24/DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6259980 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚609 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31600010934784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4229800 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 410262817792 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚28068766383676 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28982764288000 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 410037463040 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 393154654720 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788496329258034e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8704535 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚28983691495424 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7066794 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7066792 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚28590536840704 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8729381 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 249294848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1221 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚28590536840704 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 393154654720 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8704535 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 392245450240 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚32010024457728 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚28982764288000 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 225354752 _step‚25 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7066794 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚32010273752576 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848ñeãÈ'¯ÂO VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 392245450240 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6259980 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚609 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31600010934784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4229800 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 410262817792 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚28068766383676 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28982764288000 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 410037463040 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 393154654720 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788496329258034e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8704535 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚28983691495424 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7066794 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7066792 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚28590536840704 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8729381 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 249294848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1221 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚28590536840704 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 393154654720 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8704535 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 392245450240 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚32010024457728 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚28982764288000 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 225354752 _step‚25 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7066794 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚32010273752576 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2029573 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 390773677452 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28983691495424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15771327 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚28069691390492 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8729381 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1184 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚27678009745584 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚607 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 391681644908 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4229798 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15796175 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚28590518837760 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6259371 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 trainer/global_step‚349 _runtime‚ 387.038957052 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7066792 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2030180 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32101664417792 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 411198242816 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚28148901251498 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚28672155926528 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7086952 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚32101410941952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28672137923584 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6277967 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚29066432054784 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8729366 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1184 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 411427778560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 229535744 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4241865 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31690212699136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8729366 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 393366910464 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 253475840 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15841192 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚28672137923584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 FDeviceStatsMonitor.on_train_batch_end/allocÀ}êñation.all.allocated‚15841192 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2036102 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚27757010300768 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7086954 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6277348 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚31690236639232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7086952 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 394276128256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚29065504834048 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚28672155926528 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15816318 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 % _timestamp‚1.7884963304230776e+09 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 132057661449M=ÂÓ&²ÍM MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8754238 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚27757027340128 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚28149826258522 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15816318 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 393366910464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 392798918394 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 394276128256 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32101664417792 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 411198242816 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚28148901251498 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚28672155926528 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7086952 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚32101410941952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28672137923584 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6277967 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚29066432054784 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8729366 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1184 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 411427778560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 229535744 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4241865 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31690212699136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8729366 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 393366910464 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 253475840 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15841192 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚28672137923584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15841192 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2036102 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚27757010300768 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7086954 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6277348 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚31690236639232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7086952 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 394276128256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚29065504834048 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚28672155926528 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15816318 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 % _timestamp‚1.7884963304230776e+09 ODeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9974283 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9949437 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9974283 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1221 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8074792 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚32671491131904 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 448332875264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7156837 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33119806004224 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2323147 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚32671473128960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 $ _timestamp‚1.788496387757309e+09 _step‚29 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚36111272217088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8074794 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 468681076736 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚31628872433424 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚36580213074432 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8074794 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259780608 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18024229 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4833050 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚32075523365984 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚33120733211648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 446650932560 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 468916917248 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚33120733211648 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 448332875264 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 447558900016 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9949437 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚31628889472784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 _runtime‚ 445.53498472 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 M@DeviceStatsMonitor.on_train_batch_start/allocÓï#3 ation.all.allocated‚18049077 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235840512 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚36579953293824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚32076448372800 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 449242079744 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚32671491131904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0Ðîë`Ç'ÃÁO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1221 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8074792 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚32671491131904 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 448332875264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7156837 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33119806004224 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2323147 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚32671473128960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 $ _timestamp‚1.788496387757309e+09 _step‚29 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚36111272217088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8074794 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 468681076736 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚31628872433424 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚36580213074432 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8074794 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259780608 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18024229 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4833050 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚32075523365984 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚33120733211648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 446650932560 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 468916917248 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚33120733211648 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 448332875264 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 447558900016 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9949437 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚31628889472784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 _runtime‚ 445.53498472 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18049077 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235840512 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚36579953293824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚32076448372800 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 449242079744 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚32671491131904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚399 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚33119806004224 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8074792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2323787 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 449242079744 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9974283 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9949437 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9974283 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0ôþ•ì0Ä+ _wandbruntime‚446  _runtime‚446fÄuõÎ&ÅÈM I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9974268 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2328966 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18094094 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33203473771008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18069220 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4845115 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚32753092214784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 447768205838 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚31707907067328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚30 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18069220 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 450363553280 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚32156583240830 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _runtime‚ 446.714035304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7174079 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚33203473771008 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚33202546550272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚36201521861632 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18094094 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 469869216768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _runtime‚ 446.714035304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7174079 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚33203473771008 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚33202546550272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚36201521861632 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18094094 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 469869216768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2328966 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18094094 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33203473771008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18069220 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4845115 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚32753092214784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 447768205838 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚31707907067328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚30 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18069220 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 450363553280 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚32156583240830 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221160448 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 503436155124 % _timestamp‚1.7884964445046575e+09 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37257774927872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5436298 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 527560390656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚36083205355108 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36752445423104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚36752445423104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40622581379584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚33 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2616732 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35579769199984 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 504420300288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9082792 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 527339230208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37257774927872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1189 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚41149896669696 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8053030 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37256847720448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9082794 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8053645 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973ˆ#RâÍ'×ÇO YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚41149896669696 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8053030 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37256847720448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9082794 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8053645 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245100544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11219185 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36752427420160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11194339 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9082792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40622557439488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221160448 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 503436155124 % _timestamp‚1.7884964445046575e+09 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37257774927872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5436298 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 527560390656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚36083205355108 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36752445423104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚36752445423104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40622581379584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚33 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2616732 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35579769199984 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 504420300288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9082792 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 527339230208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37257774927872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1189!_•v´0Ø+ _wandbruntime‚503  _runtime‚503¾2£Ò“ ÙÍM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5448365 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235827200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37340515487232 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20346996 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9102954 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 _step‚34 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36834064508928 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8070882 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11244042 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11219170 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 506450978304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚35658769755168 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9102954 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_poŽËé+@ol.freed‚36834046505984 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8071503 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 trainer/global_step‚449 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚36834046505984 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚41241277352960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2622519 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚41241537120256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40712783143936 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 528730036224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 504553428610 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20322122 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 505541760512 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37339588266496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚ 503.212333055 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20322122 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40712807084032 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚35658786794528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚36834064508928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11219170 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5448363 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0QÊÆs×&ÚÑM WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 503645460946 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36834046505984 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8071503 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 trainer/global_step‚449 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚36834046505984 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚41241277352960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2622519 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚41241537120256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40712783143936 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 528730036224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 504553428610 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20322122 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 505541760512 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37339588266496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚ 503.212333055 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20322122 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40712807084032 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚35658786794528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚36834064508928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11219170 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5448363 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5448365 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235827200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37340515487232 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20346996 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9102954 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 _step‚34 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36834064508928 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8070882 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11244042 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11219170 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 506450978304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚35658769755168 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9102954 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 561416929792 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2911284 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 586338747904 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚40089037330600 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 225354752 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 trainer/global_step‚499 # _timestamp‚1.78849649657266e+09 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41393889436672 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10090792 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1253 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚39530648927184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1216 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22554881 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚659 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8950175 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 559313410232®:ånÏ'éÉO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚40089962337416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚657 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12439241 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 560507725312 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12464087 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚39530631887824 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22530033 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 560507725312 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 561416929792 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6039548 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45133866601984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8950834 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚41394816644096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚45133842661888 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚40833381711360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12439241 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 558405442776 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12464087 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41394816644096 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41393889436672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2910627 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 586113393152 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚40833399714304 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10090794 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 249294848 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22554881 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚659 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8950175 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 559313410232 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 561416929792 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2911284 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 586338747904 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚40089037330600 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 225354752 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 trainer/global_step‚499 # _timestamp‚1.78849649657266e+09 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41393889436672 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10090792 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1253 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚39530648927184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1216 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 555.327792276 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12488944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚40170097205446 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 560430683718 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚40915000797184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10110954 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45811366892544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45224068366336 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22599898 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8968165 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10110954 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10110952 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22599898 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 561629185536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚39609649482368 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 559522716054 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1253 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚40915018800128 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8968829 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6051615 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22575024 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 219049984 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚41477557203456 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12464072 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 % _timestamp‚1.7884964975499082e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12464072 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚40915000797184 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 561629185536 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚41476629982720 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45811609882624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 587517576192 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 587298526208 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚38 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚40915018800128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22575024 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6051613 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2916552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚499 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12488944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0[Ñ$îÕM ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22599898 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8968165 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10110954 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10110952 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22599898 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 561629185536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚39609649482368 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 559522716054 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1253 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚40915018800128 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8968829 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6051615 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22575024 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 219049984 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚41477557203456 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12464072 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 % _timestamp‚1.7884964975499082e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12464072 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚40915000797184 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 561629185536 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚41476629982720 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45811609882624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 587517576192 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 587298526208 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚38 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚40915018800128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22575024 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6051613 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2916552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚499 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12488944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 555.327792276 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12488944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on ¸ëøJ_train_batch_end/requested_bytes.all.allocated‚40170097205446 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 560430683718 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚40915000797184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10110954 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45811366892544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45224068366336&fÒD00ï+ _wandbruntime‚555  _runtime‚555' ΤOðÉ 0loss/SRqqt4/detection_loss‚2.836188633081525 /loss/FBt1/detection_loss‚1.4979544216184877 _step‚39 .loss/FRt4/assignment_loss‚6.15112912421133 /loss/FRt1/detection_loss‚2.5668590710713315 1loss/SRqqt2/detection_loss‚4.4238736689090725 .loss/FBt3/detection_loss‚4.822544193267822 epoch‚0 2loss/SRqqt1/assignment_loss‚2.8216094398498535 1loss/SRqqt1/detection_loss‚3.7899071311950685 trainer/global_step‚499 /loss/FRt1/assignment_loss‚6.635998597511878 .loss/FRt2/detection_loss‚3.112654913555492 .loss/FRt4/detection_loss‚2.669223063132342 .loss/FBt2/detection_loss‚4.975861091831965 /loss/FBt4/detection_loss‚10.154516701896986 1loss/SRqqt2/assignment_loss‚3.021209216117859 0loss/FBt4/assignment_loss‚0.6069519519805908 _runtime‚ 555.328731911 0loss/FBt1/assignment_loss‚0.2974366196859939 /loss/FRt3/detection_loss‚2.2194929405794306 1loss/SRqqt3/assignment_loss‚3.043700365736933 %loss/total_loss‚82.57300204176703 % _timestamp‚1.7884964975516999e+09 /loss/FRt2/assignment_loss‚6.611145843159068 0loss/FBt2/assignment_loss‚0.7575351968407631 1loss/SRqqt3/detection_loss‚1.0673231207719533 0loss/FRt3/assignment_loss‚5.4516813270116256 0loss/SRqqt4/assignment_loss‚2.30400279233622 0loss/FBt3/assignment_loss‚0.7342026164134339ø3y@SñÍ .loss/FBt2/detection_loss‚4.975861091831965 /loss/FBt4/detection_loss‚10.154516701896986 1loss/SRqqt2/assignment_loss‚3.021209216117859 0loss/FBt4/assignment_loss‚0.6069519519805908 _runtime‚ 555.328731911 0loss/FBt1/assignment_loss‚0.2974366196859939 /loss/FRt3/detection_loss‚2.2194929405794306 1loss/SRqqt3/assignment_loss‚3.043700365736933 %loss/total_loss‚82.57300204176703 % _timestamp‚1.7884964975516999e+09 /loss/FRt2/assignment_loss‚6.611145843159068 0loss/FBt2/assignment_loss‚0.7575351968407631 1loss/SRqqt3/detection_loss‚1.0673231207719533 0loss/FRt3/assignment_loss‚5.4516813270116256 0loss/SRqqt4/assignment_loss‚2.30400279233622 0loss/FBt3/assignment_loss‚0.7342026164134339 0loss/SRqqt4/detection_loss‚2.836188633081525 /loss/FBt1/detection_loss‚1.4979544216184877 _step‚39 .loss/FRt4/assignment_loss‚6.15112912421133 /loss/FRt1/detection_loss‚2.5668590710713315 1loss/SRqqt2/detection_loss‚4.4238736689090725 .loss/FBt3/detection_loss‚4.822544193267822 epoch‚0 2loss/SRqqt1/assignment_loss‚2.8216094398498535 1loss/SRqqt1/detection_loss‚3.7899071311950685 trainer/global_step‚499 /loss/FRt1/assignment_loss‚6.635998597511878 .loss/FRt2/detection_loss‚3.112654913555492 .loss/FRt4/detection_loss‚2.669223063132342'[8‘ò‚(:† €•éÔØ·çm gpu.0.gpu‚44.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚159.696' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.924 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥K– ó‚(:‚ €•éÔ㇪n disk.sda2.out‚62.51220703125 disk.nvme1n1.out‚ 3052.4375 disk.rbd0.out‚0 network.recv‚274151 disk.sda1.out‚0! disk.nvme0n1.out‚ 2876.1484375 disk.md0.out‚ 60.1796875 disk.sdb2.out‚62.51220703125 disk.nvme1n1.in‚ 5.83203125$ memory_percent‚98.63767623901367& proc.memory.availableMB‚ 223.203125* proc.memory.percent‚17.735981941223145 disk.rbd2.out‚0 disk.sda.in‚0 disk.dm-0.in‚ 61.18359375 proc.cpu.threads‚17) disk./.usagePercent‚83.70304472728353 disk.rbd2.in‚0# proc.memory.rssMB‚ 2905.86328125 cpu‚24.987766559982475 disk.sdb2.in‚0 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sdb.in‚0 disk.md0p1.in‚0 disk.md0.in‚0 network.sent‚733482 disk.sda1.in‚0 disk.sdb.out‚62.51220703125 disk.nvme0n1.in‚ 55.3515625% disk./.usageGB‚2993.6263160705566 disk.sda.out‚62.51220703125 disk.sdb1.in‚0 disk.md0p1.out‚ 60.1796875 disk.dm-0.out‚ 5928.5859375 disk.sda2.in‚0ŒdÑ´œô‚(:‘ •éÔÑ©Ÿm gpu.0.gpu‚31.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚117.81' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.452499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K;>Hõ‚(:Š •éÔ‡…¹m disk.sda2.in‚0 disk.sda2.out‚64.02197265625 disk.rbd2.in‚0 disk.sdb2.in‚0 disk.sdb1.out‚0 network.sent‚762222" disk.nvme1n1.out‚ 3057.16796875% disk./.usageGB‚2993.6305923461914 disk.sdb.out‚64.02197265625 disk.md0.out‚ 61.61328125 proc.cpu.threads‚17 disk.sdb.in‚0 disk.nvme1n1.in‚ 5.83203125 disk.rbd2.out‚0 disk.sdb1.in‚0# proc.memory.rssMB‚ 2905.86328125* proc.memory.percent‚17.735981941223145 network.recv‚280097 disk.sda1.out‚0 disk.rbd0.out‚0 disk.md0p1.out‚ 61.61328125 disk.dm-0.out‚ 6063.640625" disk.nvme0n1.out‚ 3006.47265625 disk.sda.out‚64.02197265625 disk.rbd0.in‚0$ memory_percent‚98.63789081573486 disk.sdb2.out‚64.02197265625 disk.sda.in‚0 disk.sda1.in‚0 disk.md0p1.in‚0 disk.dm-0.in‚ 61.18359375( proc.memory.availableMB‚ 223.16796875 cpu‚24.988962238688465) disk./.usagePercent‚83.70316429374003 disk.nvme0n1.in‚ 55.3515625 disk.md0.in‚0…á$¥“ö‚(:ˆ ž•éÔ犓m gpu.0.gpu‚29.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.891' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.72275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áêTý÷‚(:‹ ž•éÔ¬÷îm disk.rbd2.in‚0 disk.sda.out‚65.43896484375 disk.sdb1.out‚0 disk.sdb.out‚65.43896484375 network.sent‚766495 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sdb1.in‚0 disk.dm-0.out‚ 6128.86328125( proc.memory.availableMB‚ 223.41796875 cpu‚24.989591943939494) disk./.usagePercent‚83.70316429374003 disk.sda.in‚0 disk.sdb.in‚0$ memory_percent‚98.63636493682861# proc.memory.rssMB‚ 2905.86328125 disk.rbd2.out‚0 disk.md0p1.in‚0 disk.nvme0n1.in‚ 55.3515625" disk.nvme0n1.out‚ 3071.58984375 disk.nvme1n1.in‚ 5.83203125% disk./.usageGB‚2993.6305923461914 disk.md0.in‚0 disk.md0.out‚ 62.96484375 disk.sdb2.in‚0 disk.sda1.out‚0! disk.nvme1n1.out‚ 3057.2734375* proc.memory.percent‚17.735981941223145 disk.sdb2.out‚65.43896484375 disk.dm-0.in‚ 61.18359375 disk.sda2.in‚0 disk.sda2.out‚65.43896484375 proc.cpu.threads‚17 network.recv‚284908 disk.md0p1.out‚ 62.96484375k¹@²0ø+ _wandbruntime‚608  _runtime‚608‹@Äù¾ & lr-AdamW/pg1‚0.0004778678252457264 & lr-AdamW/pg2‚0.0004778678252457264 trainer/global_step‚549 % _timestamp‚1.7884965511118486e+09 _runtime‚ 608.889505168 _step‚40Ž%O Èú & lr-AdamW/pg1‚0.0004778678252457264 & lr-AdamW/pg2‚0.0004778678252457264 trainer/global_step‚549 % _timestamp‚1.7884965511118486e+09 _runtime‚ 608.889505168 _step‚40(Ä¤Ë 0û+ _wandbruntime‚608  _runtime‚608öks®Í'üÇO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13684143 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1253 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚44914354005504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 616595150336 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45530931152896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44095794312908 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 615190665340 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚44914336002560 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50290213329920 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 614282697884 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24782935 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3206282 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11098794 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13708989 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚41 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6642800 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1216 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 242131968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚50290479401984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚549 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11098792 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 617504354816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚45530931152896 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚45531858360320 % _timestamp‚1.7884965511131017e+09 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚45531858360320 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9849747 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚667 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49645151824384 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 645085445632 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13684143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚45530931152896 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚45531858360320 % _timestamp‚1.7884965511131017e+09 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚45531858360320 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9849747 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚667 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49645151824384 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 645085445632 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13684143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45530931152896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44095794312908 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 615190665340 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚44914336002560 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50290213329920 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 614282697884 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24782935 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3206282 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11098794 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13708989 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚41 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6642800 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1216 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 242131968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973)º4¾0þ+ _wandbruntime‚609  _runtime‚609²k9í´ÿÐM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24827926 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚45614598919680 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 617716610560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚45614598919680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 616307938826 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 618625828352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50381856181760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 646249117184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11118954 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚49735377528832 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 646478652928 _step‚42 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6654863 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24852800 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43560529209568 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚44995973091328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚44995955088384 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9867050 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3212187 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13733846 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11118954 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944n^YÚ&€ÔM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 618625828352 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44175929180730 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24852800 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43560529209568 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚44995973091328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚44995955088384 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9867050 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3212187 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13733846 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11118954 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24827926 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚45614598919680 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 617716610560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚45614598919680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 616307938826 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 618625828352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50381856181760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 646249117184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11118954 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚49735377528832 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 646478652928 _step‚42 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6654863 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1253 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚642 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚49667972869120 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 671067920448 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 670159952992 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 703920954368 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12106794 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10747829 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 673591779840 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221160448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12106792 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚49667972869120 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚48995290293760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12106794 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3501779 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14929045 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14953891 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3501137 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0œ~/Ó'‘ÍO Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚49667972869120 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 671067920448 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 670159952992 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 703920954368 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12106794 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10747829 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 673591779840 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221160448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12106792 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚49667972869120 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚48995290293760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12106794 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3501779 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14929045 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14953891 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3501137 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27060685 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚54860334061056 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 672682575360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12106792 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚54860579161600 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚49668900076544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245100544 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚49668900076544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1253 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚642 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144-«d0’+ _wandbruntime‚665  _runtime‚665̱‡×&“ÑM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12126952 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27105702 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27080828 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚49076909379584 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3506960 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 705328375808 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚54246662751232 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚49751640635904 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12126952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216952832 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚54951750234112 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14978748 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14953876 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 674713253376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 673804035584 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚54246638811136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚599 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12126954 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14978748 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240892928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1253 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7258113 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10765073 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚54951991127040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚48183611170062 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚48182686163038 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14953876 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚47511425976128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27105702 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚47511408936768üŽÛ&”ÕM D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚47511408936768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12126952 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27105702 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27080828 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚49076909379584 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3506960 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 705328375808 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚54246662751232 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚49751640635904 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12126952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216952832 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚54951750234112 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14978748 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14953876 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 674713253376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 673804035584 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚54246638811136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚599 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12126954 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14978748 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240892928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1253 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7258113 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10765073 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚54951991127040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚48183611170062 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚48182686163038 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14953876 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚47511425976128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27105702 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0.f*µ¡0•+ _wandbruntime‚665  _runtime‚665]óÙ–Æ /loss/SRqqt4/detection_loss‚2.93501754436228 .loss/FBt2/detection_loss‚3.462801406258031 %loss/total_loss‚69¨G"ÞÍ.96335821969483 /loss/FRt2/assignment_loss‚5.368657276034355 0loss/FBt4/assignment_loss‚0.6216536536812782 /loss/FRt1/detection_loss‚3.2028437554836273 $ _timestamp‚1.788496607877659e+09 1loss/SRqqt3/assignment_loss‚2.782693948211341 0loss/FBt1/assignment_loss‚0.4491463807946096 0loss/FBt2/assignment_loss‚0.9344320720012643 /loss/FRt2/detection_loss‚2.9942346792668104 /loss/FRt1/assignment_loss‚5.132593830426534 epoch‚0 trainer/global_step‚599 1loss/SRqqt1/assignment_loss‚2.174735437739979 0loss/FBt3/assignment_loss‚0.3377985762107528 .loss/FBt3/detection_loss‚3.947454333305359 _runtime‚ 665.654646448 _step‚47 /loss/FBt1/detection_loss‚0.8292385072650551 .loss/FBt4/detection_loss‚5.790059983730316 /loss/FRt4/assignment_loss‚5.820035537083943 .loss/FRt4/detection_loss‚2.300756784500899 1loss/SRqqt2/detection_loss‚2.9177754542657306 /loss/FRt3/assignment_loss‚5.217486914247274 1loss/SRqqt4/assignment_loss‚2.510971473322974 /loss/FRt3/detection_loss‚2.0721004982478917 0loss/SRqqt1/detection_loss‚4.226877765222029 1loss/SRqqt3/detection_loss‚1.0557684055690109 1loss/SRqqt2/assignment_loss‚2.878224002463477_ä~.P—Ê 1loss/SRqqt1/assignment_loss‚2.174735437739979 0loss/FBt3/assignment_loss‚0.3377985762107528 .loss/FBt3/detection_loss‚3.947454333305359 _runtime‚ 665.654646448 _step‚47 /loss/FBt1/detection_loss‚0.8292385072650551 .loss/FBt4/detection_loss‚5.790059983730316 /loss/FRt4/assignment_loss‚5.820035537083943 .loss/FRt4/detection_loss‚2.300756784500899 1loss/SRqqt2/detection_loss‚2.9177754542657306 /loss/FRt3/assignment_loss‚5.217486914247274 1loss/SRqqt4/assignment_loss‚2.510971473322974 /loss/FRt3/detection_loss‚2.0721004982478917 0loss/SRqqt1/detection_loss‚4.226877765222029 1loss/SRqqt3/detection_loss‚1.0557684055690109 1loss/SRqqt2/assignment_loss‚2.878224002463477 /loss/SRqqt4/detection_loss‚2.93501754436228 .loss/FBt2/detection_loss‚3.462801406258031 %loss/total_loss‚69.96335821969483 /loss/FRt2/assignment_loss‚5.368657276034355 0loss/FBt4/assignment_loss‚0.6216536536812782 /loss/FRt1/detection_loss‚3.2028437554836273 $ _timestamp‚1.788496607877659e+09 1loss/SRqqt3/assignment_loss‚2.782693948211341 0loss/FBt1/assignment_loss‚0.4491463807946096 0loss/FBt2/assignment_loss‚0.9344320720012643 /loss/FRt2/detection_loss‚2.9942346792668104 /loss/FRt1/assignment_loss‚5.132593830426534 epoch‚0 trainer/global_step‚599/åÝ˜‚(:ˆ é•éÔØô‚m gpu.0.gpu‚25.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.953' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.48825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'/© ™‚(: é•éÔ€ÐÅn disk.sda1.out‚0 disk.sdb.out‚73.74853515625 network.recv‚307866 disk.sda1.in‚0 disk.sdb.in‚0 disk.sda2.in‚0 disk.nvme1n1.in‚ 5.83203125 proc.cpu.threads‚17 disk.sda.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0! disk.nvme0n1.out‚ 3127.3046875( proc.memory.availableMB‚ 222.84765625 disk.rbd2.out‚0 disk.sda.out‚73.74853515625 disk.rbd0.in‚0# proc.memory.rssMB‚ 2905.87109375 disk.md0.in‚0 disk.md0.out‚ 70.9765625 disk.sdb2.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 61.18359375 disk.dm-0.out‚ 6879.7734375 disk.sda2.out‚73.74853515625 disk.nvme1n1.out‚ 3752.46875# memory_percent‚98.6398458480835 cpu‚24.99244053885556 disk.md0p1.out‚ 70.9765625 disk.nvme0n1.in‚ 55.3515625 network.sent‚851678 disk.rbd2.in‚0 disk.md0p1.in‚0* proc.memory.percent‚17.736029624938965( disk./.usagePercent‚83.7207087780082$ disk./.usageGB‚2994.258068084717 disk.sdb2.out‚73.74853515625[7ÛÓš‚(:’ ø•éÔ—ï®k gpu.0.gpu‚53.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚118.474' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.618500000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<ÂÚG›‚(:‡ ø•éÔŠù»l disk.sdb1.out‚0 disk.md0.in‚0 disk.sdb2.in‚0 disk.sda.in‚0 disk.sda.out‚75.17236328125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-0.out‚ 7087.18359375 disk.sdb2.out‚75.17236328125 network.sent‚887893 disk.md0p1.out‚ 72.3359375 disk.dm-0.in‚ 61.18359375 proc.cpu.threads‚17) disk./.usagePercent‚83.72413994066603 disk.sda2.out‚75.17236328125 disk.nvme0n1.in‚ 55.3515625" disk.nvme1n1.out‚ 3959.83203125 disk.rbd2.out‚0 network.recv‚313372% disk./.usageGB‚2994.3807830810547 disk.sdb1.in‚0 disk.md0.out‚ 72.3359375& proc.memory.availableMB‚ 222.828125 disk.rbd0.out‚0 disk.md0p1.in‚0 disk.sdb.out‚75.17236328125 disk.sda2.in‚0 disk.rbd2.in‚0$ memory_percent‚98.63996505737305# proc.memory.rssMB‚ 2905.87109375 disk.sda1.out‚0 disk.sdb.in‚0! disk.nvme0n1.out‚ 3127.3515625 disk.nvme1n1.in‚ 5.83203125* proc.memory.percent‚17.736029624938965 cpu‚24.993450001860523áx’œ‚(:‡ ‡–éÔØæïk gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚181.414' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.3535 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹’;Ï‚(:‡ ‡–éÔÙÍm network.recv‚317544% disk./.usageGB‚2994.6318016052246 disk.md0.in‚0 disk.sdb.in‚0 disk.sdb.out‚76.65966796875 disk.nvme0n1.in‚ 55.3515625) disk./.usagePercent‚83.73115852366124 disk.sdb2.in‚0 disk.sdb2.out‚76.65966796875 disk.sda1.in‚0 disk.sda.out‚76.65966796875 disk.rbd0.in‚0# proc.memory.rssMB‚ 2905.87109375 proc.cpu.threads‚17" disk.nvme1n1.out‚ 4129.98046875 disk.md0p1.out‚ 73.7578125' proc.memory.availableMB‚ 223.0390625* proc.memory.percent‚17.736029624938965 disk.rbd2.in‚0 disk.dm-0.out‚ 7257.4765625 cpu‚24.993846603933992 disk.nvme1n1.in‚ 5.83203125 disk.md0.out‚ 73.7578125 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sdb1.in‚0 disk.md0p1.in‚0 disk.dm-0.in‚ 61.18359375" disk.nvme0n1.out‚ 3127.49609375 network.sent‚895864 disk.sda2.in‚0# memory_percent‚98.6386775970459 disk.rbd2.out‚0 disk.sda.in‚0 disk.sdb1.out‚0 disk.sda2.out‚76.65966796875R¾³F‘ž‚(:† ––éÔÃôðk gpu.0.gpu‚34.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.344' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kª.ÓŸ‚(:ƒ ––éÔ‹èŒm disk.sda.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0% disk./.usageGB‚2994.6307830810547 disk.sda1.in‚0$ memory_percent‚98.63951206207275( proc.memory.availableMB‚ 222.90234375* proc.memory.percent‚17.736029624938965 disk.sdb1.out‚0 disk.dm-0.out‚ 7467.73046875 disk.nvme0n1.in‚ 55.3515625 disk.nvme1n1.in‚ 5.83203125 disk.sdb2.out‚ 78.5888671875 disk.sda.out‚ 78.5888671875 disk.rbd0.in‚0 disk.sda2.out‚ 78.5888671875 network.recv‚322483 disk.sdb1.in‚0 disk.md0p1.out‚ 75.609375 disk.sdb.in‚0# proc.memory.rssMB‚ 2905.87109375 cpu‚24.994675649092894" disk.nvme0n1.out‚ 3127.70703125 disk.md0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚0) disk./.usagePercent‚83.73113004529915 disk.dm-0.in‚ 61.18359375 disk.sda1.out‚0 network.sent‚900040 disk.md0.out‚ 75.609375 disk.md0p1.in‚0! disk.nvme1n1.out‚ 4340.0234375 disk.sdb2.in‚0 disk.rbd0.out‚0 disk.sdb.out‚ 78.5888671875æÿ«0 + _wandbruntime‚721  _runtime‚721^#¢Ä¡¾ & lr-AdamW/pg1‚0.0005649111449626164 & lr-AdamW/pg2‚0.0005649111449626164 trainer/global_step‚649 % _timestamp‚1.7884966636689036e+09 _runtime‚ 721.446156369 _step‚48´‘•õȢ & lr-AdamW/pg1‚0.0005649111449626164 & lr-AdamW/pg2‚0.0005649111449626164 trainer/global_step‚649 % _timestamp‚1.7884966636689036e+09 _runtime‚ 721.446156369 _step‚480Iâ 40£+ _wandbruntime‚721  _runtime‚721«D\Ï'¤ÉO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3797048 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚59430814991872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 762863173632 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13114794 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16198793 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13114792 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚53805014585344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1253 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 729679204864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16173947 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 253489152 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚52109308277524 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 729679204864 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884966636696708e+09 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 728770000384 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7849298 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 763092722688 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7849300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1216 trainer/global_step‚649 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29288739 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚53805941792768 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13114792 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58667722269184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640‘¤sÓ'¥ÍO _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚51383288108784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚665 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 728770000384 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚ 721.447031393 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚667 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 726037208100 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚53076262587904 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚53076244584960 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7849298 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 763092722688 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7849300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1216 trainer/global_step‚649 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29288739 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚53805941792768 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13114792 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58667722269184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3797048 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚59430814991872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 762863173632 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13114794 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16198793 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13114792 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚53805014585344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1253 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 729679204864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16173947 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 253489152 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚52109308277524 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 729679204864 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884966636696708e+09 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 728770000384 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13134954 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13134952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11663595 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7861363 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 764044815360 _step‚50 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16198778 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 247184384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚521táÂ89443145346 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚53888682352128 _runtime‚ 722.453023808 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚53887755131392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16223650 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53157863670784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13134954 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚649 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 223244288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 729891460608 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13134952 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7861365 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29333730 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚59521968848896 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 730800678400 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58757947973632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29358604 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16198778 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚53157863670784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3802890 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚52190368152370 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53157881673728 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 $ _timestamp‚1.788496664675133e+09 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚51462288663968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 729891460608 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29333730 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11664255 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚660 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚658 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1216 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚59522216033280 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58757924033536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16223650 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13134954 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13134952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11663595 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7861363 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 764044815360 _step‚50 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16198778 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 247184384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚52189443145346 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚53888682352128 _runtime‚ 722.453023808 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚53887755131392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16223650 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53157863670784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13134954 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚649 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 223244288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 729891460608 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13134952 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7861365 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29333730 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚59521968848896 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 730800678400 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58757947973632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29358604 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16198778 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚53157863670784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3802890 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚52190368152370 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53157881673728 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 $ _timestamp‚1.788496664675133e+09 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚51462288663968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 729891460608 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29333730 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11664255 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚660 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚658 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1216 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚59522216033280 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58757924033536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚6152:t§!0©+ _wandbruntime‚722  _runtime‚722)@ˆNªÈ /loss/FRt3/assignment_loss‚5.451530321784642 1loss/SRqqt2/assignment_loss‚2.724403310705114 1loss/FBt1/assignment_loss‚0.37316648082022486 0loss/SRqqt4/detection_loss‚2.215500776966413 epoch‚0 /loss/FRt2/assignment_loss‚5.850826796363382 2loss/SRqqt1/assignment_loss‚2.4752513647079466 0loss/FBt4/assignment_loss‚1.1819307208061218 1loss/SRqqt1/detection_loss‚2.8619631469249724 /loss/FBt1/detection_loss‚1.0703654155353644 /loss/FBt3/detection_loss‚3.6125173966089883 .loss/FBt4/detection_loss‚5.117984771728516 _step‚51 /loss/FBt3/assignment_loss‚0.693430620763037 0loss/SRqqt2/detection_loss‚3.017017643760752 2loss/SRqqt4/assignment_loss‚2.6925240866839886 .loss/FRt1/detection_loss‚2.190825391106489 -loss/FRt2/detection_loss‚2.79260821202222 0loss/SRqqt3/detection_loss‚1.009864720796782 trainer/global_step‚649 _runtime‚ 722.455214967 0loss/FBt2/assignment_loss‚0.7094217269840166 2loss/SRqqt3/assignment_loss‚2.9455380969577365 -loss/FRt4/detection_loss‚2.32606653124094 % _timestamp‚1.7884966646782296e+09 0loss/FRt4/assignment_loss‚5.8626662492752075 0loss/FRt1/assignment_loss‚3.9582853549864234 /loss/FRt3/detection_loss‚2.5159943220408065 -loss/FBt2/detection_loss‚3.52652422445161 %loss/total_loss‚67.17620768402169þ8¼âR«Ì -loss/FBt2/detection_loss‚3.52652422445161 %loss/total_loss‚67.17620768402169 /loss/FRt3/assignment_loss‚5.451530321784642 1loss/SRqqt2/assignment_loss‚2.724403310705114 1loss/FBt1/assignment_loss‚0.37316648082022486 0loss/SRqqt4/detection_loss‚2.215500776966413 epoch‚0 /loss/FRt2/assignment_loss‚5.850826796363382 2loss/SRqqt1/assignment_loss‚2.4752513647079466 0loss/FBt4/assignment_loss‚1.1819307208061218 1loss/SRqqt1/detection_loss‚2.8619631469249724 /loss/FBt1/detection_loss‚1.0703654155353644 /loss/FBt3/detection_loss‚3.6125173966089883 .loss/FBt4/detection_loss‚5.117984771728516 _step‚51 /loss/FBt3/assignment_loss‚0.693430620763037 0loss/SRqqt2/detection_loss‚3.017017643760752 2loss/SRqqt4/assignment_loss‚2.6925240866839886 .loss/FRt1/detection_loss‚2.190825391106489 -loss/FRt2/detection_loss‚2.79260821202222 0loss/SRqqt3/detection_loss‚1.009864720796782 trainer/global_step‚649 _runtime‚ 722.455214967 0loss/FBt2/assignment_loss‚0.7094217269840166 2loss/SRqqt3/assignment_loss‚2.9455380969577365 -loss/FRt4/detection_loss‚2.32606653124094 % _timestamp‚1.7884966646782296e+09 0loss/FRt4/assignment_loss‚5.8626662492752075 0loss/FRt1/assignment_loss‚3.9582853549864234 /loss/FRt3/detection_loss‚2.51599432204080653ŒÆú‡“¬‚(:ˆ ¥–éÔªöÜm gpu.0.gpu‚36.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hHËé­‚(:‰ ¥–éÔË®ˆn disk.sdb.in‚0$ memory_percent‚98.63765239715576( proc.memory.availableMB‚ 223.20703125# proc.memory.rssMB‚ 2905.87109375 cpu‚24.995372601028016 network.sent‚905205 network.recv‚326007% disk./.usageGB‚2994.6307830810547 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚80.14501953125 proc.cpu.threads‚17) disk./.usagePercent‚83.73113004529915 disk.rbd0.out‚0 disk.nvme1n1.in‚ 5.83203125 disk.md0.out‚ 77.0859375 disk.sdb1.out‚0 disk.md0p1.out‚ 77.0859375 disk.dm-0.in‚ 61.18359375 disk.sdb2.in‚0 disk.sda.in‚0 disk.sda.out‚80.14501953125 disk.dm-0.out‚ 7683.33984375 disk.nvme0n1.in‚ 55.3515625 disk.md0.in‚0 disk.sdb2.out‚80.14501953125 disk.md0p1.in‚0! disk.nvme0n1.out‚ 3128.2265625" disk.nvme1n1.out‚ 4555.11328125 disk.rbd2.out‚0 disk.sda1.in‚0 disk.sdb.out‚80.14501953125 disk.rbd2.in‚0* proc.memory.percent‚17.736029624938965 disk.rbd0.in‚0 disk.sdb1.in‚0Z|Y®‚(:‰ ´–éÔ·‚úl% disk./.usageGB‚2994.6307830810547 disk.dm-0.in‚ 61.18359375 disk.sdb.out‚81.79248046875 disk.sdb2.in‚0 disk.sda.in‚0 disk.sda.out‚81.79248046875 disk.rbd0.out‚0 disk.md0p1.out‚ 78.6640625 disk.sdb1.in‚0 disk.dm-0.out‚ 8055.80859375! disk.nvme0n1.out‚ 3128.2734375 disk.rbd2.out‚0 disk.nvme0n1.in‚ 55.3515625 disk.sda1.in‚0) disk./.usagePercent‚83.73113004529915 disk.sda2.in‚0 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚0$ memory_percent‚98.63812923431396 proc.cpu.threads‚17 network.sent‚954320 network.recv‚331570 disk.sdb.in‚0 disk.md0.in‚0* proc.memory.percent‚17.736029624938965 cpu‚24.996181310164978 disk.sdb2.out‚81.79248046875 disk.rbd0.in‚0( proc.memory.availableMB‚ 223.12890625 disk.sda2.out‚81.79248046875# proc.memory.rssMB‚ 2905.87109375 disk.nvme1n1.in‚ 5.83203125" disk.nvme1n1.out‚ 4927.53515625 disk.md0.out‚ 78.6640625 disk.sda1.out‚0áÿ÷/“¯‚(:ˆ ´–éԥȯm gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&rm¿°‚(:… ÖéÔ†‹Àm gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚178.02' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.505 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙþÜÿ±‚(:ô ÖéÔÉÓÌm" disk.nvme0n1.out‚ 3130.08203125 cpu‚24.996527592633342 disk.sdb2.in‚0 disk.md0p1.in‚0# memory_percent‚98.6424446105957& proc.memory.availableMB‚ 222.421875 proc.cpu.threads‚17 disk.dm-0.out‚ 8147.296875 disk.sdb.in‚0 disk.nvme0n1.in‚ 56.55859375" disk.nvme1n1.out‚ 5017.21484375 network.sent‚960036 disk.sda.out‚ 83.2578125 disk.sda2.in‚0 disk.sda2.out‚ 83.2578125# proc.memory.rssMB‚ 2905.87890625* proc.memory.percent‚17.736077308654785 network.recv‚335439 disk.sda1.out‚0 disk.md0p1.out‚ 80.06640625 disk.nvme1n1.in‚ 5.83203125 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 62.390625 disk.md0.in‚0% disk./.usageGB‚2993.0470123291016 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 80.06640625 disk.rbd2.out‚0 disk.sdb2.out‚ 83.2578125 disk.sda.in‚0 disk.sdb.out‚ 83.2578125' disk./.usagePercent‚83.686847152215 disk.sda1.in‚0 disk.rbd0.out‚0˜­„Ä0²+ _wandbruntime‚775  _runtime‚775G‚ÒYij¾ _step‚52 & lr-AdamW/pg1‚0.0006084328048210615 & lr-AdamW/pg2‚0.0006084328048210615 trainer/global_step‚699 % _timestamp‚1.7884967178124943e+09 _runtime‚ 775.589738992}Ìæ‹È´Â _step‚52 & lr-AdamW/pg1‚0.0006084328048210615 & lr-AdamW/pg2‚0.0006084328048210615 trainer/global_step‚699 % _timestamp‚1.7884967178124943e+09 _runtime‚ 775.5897389924⛚0µ+ _wandbruntime‚775  _runtime‚775J&ÌÈY!¶ÈO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14122794 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8452550 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚55334167835984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57942056301568 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚57942056301568 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12544821 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 821927683584 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12545497 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1259 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4092273 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 785766629888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 _runtime‚ 775.590893097 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚55334150796624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64000935175168 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14122792 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 785766629888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 243003392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17418849 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚56116990266648 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17418849 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 219063296 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31541641 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17443695 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚63178983551488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚56116065259832 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17443695 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 784857425408 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 784857425408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 781914463208 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1222 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63179007491584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57942983508992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚699 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚57157216879104 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31566489 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 $ _timestamp‚1.788496717813553e+09 \IDeviceStatsMonitorš¯þðu.on_train_batch_start/active_bytes.large_pool.allocated‚57157216879104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚57157198876160 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚57157198876160 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4092947 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚64000692171776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚53 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57942983508992èì7oÒ'·ÌO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 785766629888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 _runtime‚ 775.590893097 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚55334150796624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64000935175168 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14122792 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 785766629888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 243003392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17418849 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚56116990266648 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17418849 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 219063296 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31541641 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17443695 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚63178983551488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚56116065259832 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17443695 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 784857425408 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 784857425408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 781914463208 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1222 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63179007491584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57942983508992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚699 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚57157216879104 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31566489 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 $ _timestamp‚1.788496717813553e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚57157216879104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚57157198876160 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚57157198876160 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4092947 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚64000692171776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚53 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57942983508992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14122794 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8452550 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚55334167835984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57942056301568 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚57942056301568 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12544821 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 821927683584 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12545497 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1259 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4092273 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 132057661445ë;Ñ0¸+ _wandbruntime‚776  _runtime‚776€N•YÙ&¹ÓM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14142954 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8464613 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1222 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 786888103424 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14142952 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 822881187328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚55413185430528 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31611506 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚699 _step‚54 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚57238835964928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 783939704150 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 783031736486 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31611506 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚58025724068352 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 785978885632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 225341440 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56197125134678 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17443680 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚64092339724800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚57238817961984 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚64092090443264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12562855 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 % _timestamp‚1.7884967187581809e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14142954 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17468552 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 785978885632 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12563522 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _runtime‚ 776.535828357 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31586632 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚63269233196032 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8464615 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 823106528768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14142952 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚58024796847616 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚56196200127654 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44•@{íÝ&º×M C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14142952 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚58024796847616 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚56196200127654 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14142954 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8464613 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1222 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 786888103424 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14142952 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 822881187328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚55413185430528 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31611506 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚699 _step‚54 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚57238835964928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 783939704150 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 783031736486 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31611506 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚58025724068352 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 785978885632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 225341440 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56197125134678 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17443680 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚64092339724800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚57238817961984 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚64092090443264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12562855 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 % _timestamp‚1.7884967187581809e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14142954 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17468552 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 785978885632 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12563522 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _runtime‚ 776.535828357 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31586632 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚63269233196032 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8464615 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8231065287686D&ˆj0»+ _wandbruntime‚776  _runtime‚776p­—‹r¼Å 1loss/SRqqt3/assignment_loss‚2.629709406112268 .loss/FRt1/detection_loss‚2.465718897906217 .loss/FRt4/detection_loss‚2.420719467030436 .loss/FBt1/detection_loss‚0.884075254392821 /loss/FBt4/detection_loss‚3.7378713687260947 _runtime‚ 776.537394975 1loss/FBt4/assignment_loss‚0.18251011551668248 trainer/global_step‚699 /loss/FRt1/assignment_loss‚3.849939939650622 /loss/SRqqt4/detection_loss‚2.41655249266248 ,loss/FBt2/detection_loss‚3.1009087651968 1loss/SRqqt1/assignment_loss‚2.711772396014287 0loss/FBt1/assignment_loss‚0.2634025833369121 .loss/FBt3/detection_loss‚4.818715453147888 0loss/FRt2/assignment_loss‚5.3869575170370245 /loss/FRt3/assignment_loss‚4.856788566479316 /loss/FRt4/assignment_loss‚5.455542583091586 2loss/SRqqt2/assignment_loss‚2.7360524783531823 epoch‚0 2loss/SRqqt4/as>ivÙsignment_loss‚2.8658272059340226 /loss/FRt3/detection_loss‚2.2782113288457575 1loss/SRqqt3/detection_loss‚0.9579851464103393 %loss/total_loss‚64.67528486118063 $ _timestamp‚1.788496718760411e+09 _step‚55 /loss/FRt2/detection_loss‚3.1599988157932577 0loss/FBt2/assignment_loss‚0.7934064760804176 /loss/FBt3/assignment_loss‚0.283062926599996 0loss/SRqqt1/detection_loss‚3.412937100117023 0loss/SRqqt2/detection_loss‚3.006618576745192Ò•ãNO½É /loss/FRt2/detection_loss‚3.1599988157932577 0loss/FBt2/assignment_loss‚0.7934064760804176 /loss/FBt3/assignment_loss‚0.283062926599996 0loss/SRqqt1/detection_loss‚3.412937100117023 0loss/SRqqt2/detection_loss‚3.006618576745192 1loss/SRqqt3/assignment_loss‚2.629709406112268 .loss/FRt1/detection_loss‚2.465718897906217 .loss/FRt4/detection_loss‚2.420719467030436 .loss/FBt1/detection_loss‚0.884075254392821 /loss/FBt4/detection_loss‚3.7378713687260947 _runtime‚ 776.537394975 1loss/FBt4/assignment_loss‚0.18251011551668248 trainer/global_step‚699 /loss/FRt1/assignment_loss‚3.849939939650622 /loss/SRqqt4/detection_loss‚2.41655249266248 ,loss/FBt2/detection_loss‚3.1009087651968 1loss/SRqqt1/assignment_loss‚2.711772396014287 0loss/FBt1/assignment_loss‚0.2634025833369121 .loss/FBt3/detection_loss‚4.818715453147888 0loss/FRt2/assignment_loss‚5.3869575170370245 /loss/FRt3/assignment_loss‚4.856788566479316 /loss/FRt4/assignment_loss‚5.455542583091586 2loss/SRqqt2/assignment_loss‚2.7360524783531823 epoch‚0 2loss/SRqqt4/assignment_loss‚2.8658272059340226 /loss/FRt3/detection_loss‚2.2782113288457575 1loss/SRqqt3/detection_loss‚0.9579851464103393 %loss/total_loss‚64.67528486118063 $ _timestamp‚1.788496718760411e+09 _step‚557ÐæÕý“¾‚(:ˆ Ò–éÔòªk gpu.0.gpu‚11.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.071' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.26775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ttŸ€¿‚(:‡ Ò–éÔ¶½¼m disk.sdb1.in‚0 disk.sdb.out‚85.31298828125 cpu‚24.996911447533268 disk.sda2.out‚85.31298828125 network.sent‚966222 disk.sda.out‚85.31298828125 disk.sda1.out‚0 disk.sdb1.out‚0* proc.memory.percent‚17.736077308654785 disk.md0.out‚ 82.0546875 disk.rbd2.out‚0 disk.sda.in‚0( proc.memory.availableMB‚ 222.74609375% disk./.usageGB‚2993.0468101501465 disk.md0p1.out‚ 82.0546875 disk.nvme0n1.in‚ 56.55859375$ memory_percent‚98.64046573638916 disk.sda1.in‚0 disk.md0p1.in‚0# proc.memory.rssMB‚ 2905.87890625 proc.cpu.threads‚17 disk.sda2.in‚0 disk.md0.in‚0 disk.rbd2.in‚0 disk.sdb2.in‚0 disk.dm-0.out‚ 8291.23828125 network.recv‚340331" disk.nvme1n1.out‚ 5160.31640625 disk.sdb2.out‚85.31298828125 disk.dm-0.in‚ 62.390625 disk.nvme1n1.in‚ 5.83203125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sdb.in‚0 disk.nvme0n1.out‚ 3130.921875) disk./.usagePercent‚83.68684149920679!¡{À‚(:’ á–éÔ”œ‰l gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚127.106' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚31.776499999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×AM` Á‚(:€ á–éÔÍë‰m disk.md0p1.out‚83.4375 disk.dm-0.out‚ 8294.87109375 disk.sda2.out‚86.75732421875 disk.rbd2.out‚0 proc.cpu.threads‚17 disk.sda2.in‚0" disk.nvme0n1.out‚ 3132.68359375 disk.sdb2.in‚0 network.sent‚ 1.015233e+06( disk./.usagePercent‚83.6852583369285 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sdb2.out‚86.75732421875 disk.sda1.in‚0$ memory_percent‚98.64177703857422 cpu‚24.997297146270977 network.recv‚344737$ disk./.usageGB‚2992.990188598633 disk.sdb1.in‚0 disk.nvme0n1.in‚ 56.55859375 disk.nvme1n1.in‚ 5.83203125 disk.sda.out‚86.75732421875 disk.sda1.out‚0% proc.memory.availableMB‚ 222.53125 disk.rbd0.out‚0* proc.memory.percent‚17.736077308654785 disk.sdb1.out‚0 disk.nvme1n1.out‚ 5162.1875 disk.md0.in‚0# proc.memory.rssMB‚ 2905.87890625 disk.dm-0.in‚ 62.390625 disk.md0.out‚83.4375 disk.md0p1.in‚0 disk.sdb.in‚0 disk.sdb.out‚86.75732421875 disk.sda.in‚0á|_’‚(:‡ ð–éÔ Òþl gpu.0.gpu‚58.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚132.122' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.0305 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤^̥Â(: ð–éÔõã»m disk.md0p1.out‚ 84.80859375( disk./.usagePercent‚83.6871268161302 disk.rbd2.in‚0 disk.sda.in‚0$ memory_percent‚98.64022731781006 disk.sda2.in‚0 disk.md0.out‚ 84.80859375 disk.rbd0.out‚0$ disk./.usageGB‚2993.057014465332 disk.nvme0n1.in‚ 56.55859375 disk.sdb2.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 network.sent‚ 1.020379e+06" disk.nvme0n1.out‚ 3132.74609375 disk.sdb2.out‚88.18505859375 disk.md0p1.in‚0# proc.memory.rssMB‚ 2905.87890625* proc.memory.percent‚17.736077308654785 disk.dm-0.in‚ 62.390625 disk.sdb.out‚88.18505859375 disk.rbd2.out‚0 proc.cpu.threads‚17 disk.dm-0.out‚ 8382.2890625 disk.sdb.in‚0 disk.nvme1n1.in‚ 5.83203125 disk.md0.in‚0 disk.sda.out‚88.18505859375 disk.sda1.in‚0 disk.sdb1.out‚0( proc.memory.availableMB‚ 222.78515625" disk.nvme1n1.out‚ 5249.54296875 disk.sdb1.in‚0 cpu‚24.99762062684219 network.recv‚348293 disk.sda2.out‚88.185058593753s}’Ä‚(:‡ ÿ–éÔÆúØk gpu.0.gpu‚51.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.266' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.3165 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûZ¶óÅ‚(:Ž ÿ–éÔšÂþl disk.sda1.out‚0 disk.rbd0.in‚0* proc.memory.percent‚17.736077308654785 disk.dm-0.in‚ 62.390625 disk.sdb2.in‚0" disk.nvme1n1.out‚ 5429.55859375 disk.md0p1.out‚ 86.69921875 disk.sda2.out‚90.15673828125# proc.memory.rssMB‚ 2905.87890625 disk.sdb2.out‚90.15673828125 disk.md0p1.in‚0' proc.memory.availableMB‚ 222.5234375 proc.cpu.threads‚17 disk.rbd2.out‚0) disk./.usagePercent‚83.69258282232983 disk.md0.in‚0 disk.md0.out‚ 86.69921875 disk.rbd0.out‚0 disk.dm-0.out‚ 8562.85546875 disk.sdb.in‚0 cpu‚24.997986086269357 disk.sdb.out‚90.15673828125$ memory_percent‚98.64182472229004 network.recv‚352459% disk./.usageGB‚2993.2521476745605 disk.nvme1n1.in‚ 5.83203125 disk.sda.out‚90.15673828125 disk.sdb1.in‚0 network.sent‚ 1.025519e+06 disk.sda1.in‚0 disk.rbd2.in‚0 disk.sda.in‚0 disk.sdb1.out‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 56.55859375 disk.nvme0n1.out‚ 3133.296875íÙÇ0Æ+ _wandbruntime‚828  _runtime‚828˨þÄǾ & lr-AdamW/pg1‚0.0006519544646795065 & lr-AdamW/pg2‚0.0006519544646795065 trainer/global_step‚749 % _timestamp‚1.7884967706094012e+09 _runtime‚ 828.386714576 _step‚56$ÐA[ÈÈ _runtime‚ 828.386714576 _step‚56 & lr-AdamW/pg1‚0.0006519544646795065 & lr-AdamW/pg2‚0.0006519544646795065 trainer/global_step‚749 % _timestamp‚1.7884967706094012e+098=¸ó=0É+ _wandbruntime‚828  _runtime‚828ÉÐ÷üÏ'ÊÉO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚683 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚61238153167360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚61238171170304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13445120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15130792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 255586304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 trainer/global_step‚749 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9055800 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18688597 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚62079098017792 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚61238171170304 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15130792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67690292713984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67690268773888 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚61238153167360 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚68570991342592 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚60123747248956 _runtime‚ 828.387973052 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 840944850432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚62079098017792 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 840944850432 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4388639 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚60122822242140 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 841854054912 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 841854054912 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33819391 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 880722568704 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 837791718316 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚681 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1222 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚68571246928896 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 880954214912 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 231646208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 % _timestamp‚1.7884967706105647e+09 _step‚57 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚59285030523824 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456ó ù¹Ó'ËÍO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚61238153167360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚61238171170304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13445120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15130792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 255586304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 trainer/global_step‚749 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9055800 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18688597 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚62079098017792 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚61238171170304 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15130792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67690292713984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67690268773888 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚61238153167360 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚68570991342592 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚60123747248956 _runtime‚ 828.387973052 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 840944850432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚62079098017792 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 840944850432 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4388639 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚60122822242140 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 841854054912 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 841854054912 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33819391 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 880722568704 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 837791718316 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚681 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1222 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚68571246928896 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 880954214912 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 231646208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 % _timestamp‚1.7884967706105647e+09 _step‚57 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚59285030523824 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚6839C(úŠ0Ì+ _wandbruntime‚829  _runtime‚829é÷ã>úÍÓM R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚62161838563840 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18713454 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚59364065157728 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18713454 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 842975528448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13462963 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33839534 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 839816959258 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1259 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67780494478336 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 881894799872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚68662389278208 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚61319772253184 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 842066310656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884967715738654e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 _runtime‚ 829.351810827 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reser}.´áßved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33864408 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 838908991594 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33864408 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1222 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 842066310656 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚62162765784576 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 251378688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 227438592 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33839534 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9067865 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13462297 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 882122238464 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 _step‚58 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9067863 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚59364048118368 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15150952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18688582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚61319772253184 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚61319790256128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚61319790256128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18688582 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚60203882116986 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 842975528448 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚68662640656896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4395098 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15150954 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15150952 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 842975528448 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚68662640656896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4395098 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15150954 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15150952 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18713454 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚59364065157728 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18713454 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 842975528448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13462963 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33839534 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 839816959258 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1259 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67780494478336 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 881894799872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚68662389278208 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚61319772253184 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 842066310656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884967715738654e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 _runtime‚ 829.351810827 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33864408 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 838908991594 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33864408 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1222 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 842066310656 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚62162765784576 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 251378688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 227438592 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33839534 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9067865 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13462297 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 882122238464 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 _step‚58 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9067863 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚59364048118368 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15150952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18688582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚61319772253184 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚61319790256128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0:ì5q10Ï+ _wandbruntime‚829  _runtime‚8293ClBм 1loss/SRqqt2/assignment_loss‚3.109583870569865 .loss/FRt3/detection_loss‚1.998787990771234 /loss/FBt1/detection_loss‚0.7700847289379935 epoch‚0 0loss/FRt1/assignment_loss‚3.8111153437245275 2loss/SRqqt3/assignment_loss‚2.6035558859507244 % _timestamp‚1.7884967715757961e+09 /loss/FRt2/assignment_loss‚5.938810427983602 /loss/FRt4/assignment_loss‚5.702973828315735 1loss/FBt1/assignment_loss‚0.25160598106445026 0loss/FBt3/assignment_loss‚0.9853939585387707 0loss/SRqqt1/detection_loss‚3.466644869910346 1loss/SRqqt2/detection_loss‚2.6420425295829775 loss/FBt4/assignment_loss‚0 _step‚59 2loss/SRqqt4/assignment_loss‚2.8491886156361277 /loss/FRt4/detection_loss‚2.3646367064118383 /loss/FBt2/detection_loss‚2.4692983491854235 2loss/SRqqt1/assignment_loss‚2.6969076792399087 /loss/FRt2/detection_loss‚3.2075559298197427 1loss/SRqqt4/detection_loss‚2.2655421219220977 trainer/global_step‚749 0loss/FBt2/assignment_loss‚0.6693156524138018 /loss/FRt1/detection_loss‚2.4793609649904313 1loss/SRqqt3/detection_loss‚0.9299218180278938 .loss/FBt4/detection_loss‚2.282468318939209 -loss/FRt3/assignment_loss‚4.2436110265553 /loss/FBt3/detection_loss‚3.3379284620285032 $loss/total_loss‚61.0763350605205 _runtime‚ 829.352789652øØ—FÑÀ 0loss/FRt1/assignment_loss‚3.8111153437245275 2loss/SRqqt3/assignment_loss‚2.6035558859507244 % _timestamp‚1.7884967715757961e+09 /loss/FRt2/assignment_loss‚5.938810427983602 /loss/FRt4/assignment_loss‚5.702973828315735 1loss/FBt1/assignment_loss‚0.25160598106445026 0loss/FBt3/assignment_loss‚0.9853939585387707 0loss/SRqqt1/detection_loss‚3.466644869910346 1loss/SRqqt2/detection_loss‚2.6420425295829775 loss/FBt4/assignment_loss‚0 _step‚59 2loss/SRqqt4/assignment_loss‚2.8491886156361277 /loss/FRt4/detection_loss‚2.3646367064118383 /loss/FBt2/detection_loss‚2.4692983491854235 2loss/SRqqt1/assignment_loss‚2.6969076792399087 /loss/FRt2/detection_loss‚3.2075559298197427 1loss/SRqqt4/detection_loss‚2.2655421219220977 trainer/global_step‚749 0loss/FBt2/assignment_loss‚0.6693156524138018 /loss/FRt1/detection_loss‚2.4793609649904313 1loss/SRqqt3/detection_loss‚0.9299218180278938 .loss/FBt4/detection_loss‚2.282468318939209 -loss/FRt3/assignment_loss‚4.2436110265553 /loss/FBt3/detection_loss‚3.3379284620285032 $loss/total_loss‚61.0763350605205 _runtime‚ 829.352789652 1loss/SRqqt2/assignment_loss‚3.109583870569865 .loss/FRt3/detection_loss‚1.998787990771234 /loss/FBt1/detection_loss‚0.7700847289379935 epoch‚0;뚤ÁÒ‚(:… Ž—éÔ¬”ˆl gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.512' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òZGåÓ‚(:Š Ž—éÔݺ–m$ disk./.usageGB‚2993.381134033203 disk.sda1.in‚0 disk.md0p1.out‚ 88.28515625 disk.sdb.in‚0 disk.sda2.out‚91.80322265625! disk.nvme0n1.out‚ 3133.3359375 disk.rbd2.out‚0 disk.sdb2.in‚0 disk.sda.out‚91.80322265625 disk.sdb1.in‚0 disk.dm-0.in‚ 62.390625 network.sent‚ 1.031755e+06) disk./.usagePercent‚83.69618933490247 disk.sda.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1n1.in‚ 5.83203125$ memory_percent‚98.64108562469482 disk.md0.in‚0 disk.nvme0n1.in‚ 56.55859375( proc.memory.availableMB‚ 222.64453125 disk.md0.out‚ 88.28515625 disk.rbd2.in‚0 disk.sdb2.out‚91.80322265625 disk.sdb.out‚91.80322265625 network.recv‚356999 disk.sdb1.out‚0 disk.md0p1.in‚0 disk.sda2.in‚0! disk.nvme1n1.out‚ 5531.6796875" proc.memory.rssMB‚ 2905.8828125* proc.memory.percent‚17.736101150512695 cpu‚24.99829933928372 disk.dm-0.out‚ 8665.015625 proc.cpu.threads‚17Ь×f‘Ô‚(:† —éÔûñâk gpu.0.gpu‚20.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚123.712' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.928 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿žˆ-Õ‚(:‰ —éÔõ‡ål network.recv‚361089 disk.md0p1.out‚ 90.140625 disk.sda2.out‚93.73095703125 disk.md0.in‚0 disk.md0.out‚ 90.140625 disk.rbd2.in‚0 disk.sda.out‚93.73095703125" proc.memory.rssMB‚ 2905.8828125) disk./.usagePercent‚83.70670872988298 disk.sda2.in‚0 disk.sdb2.in‚0 disk.sda.in‚0# memory_percent‚98.6407995223999( proc.memory.availableMB‚ 222.69140625 cpu‚24.99891810547933 disk.sdb1.out‚0 disk.md0p1.in‚0 disk.dm-0.out‚ 8782.81640625 disk.sdb.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.736101150512695% disk./.usageGB‚2993.7573585510254 disk.sdb2.out‚93.73095703125 disk.sdb1.in‚0 network.sent‚ 1.079941e+06 disk.sdb.out‚93.73095703125 disk.nvme0n1.in‚ 56.55859375" disk.nvme0n1.out‚ 3133.51171875 disk.nvme1n1.in‚ 5.83203125 disk.rbd2.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 62.390625! disk.nvme1n1.out‚ 5649.3046875môÛ‘Ö‚(:† ¬—éÔî©Ïk gpu.0.gpu‚44.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.916' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.229 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”¤\ׂ(:õ ¬—éÔœÖm network.recv‚365605 disk.sdb1.in‚0 network.sent‚ 1.085851e+06 disk.rbd2.out‚0 disk.sda1.in‚0 disk.md0p1.out‚91.4375 disk.dm-0.in‚ 62.390625 disk.nvme1n1.in‚ 5.83203125! disk.nvme1n1.out‚ 5797.1015625" proc.memory.rssMB‚ 2905.8828125 disk.md0.out‚91.4375 disk.sdb2.out‚ 95.0859375 disk.rbd0.in‚0 disk.sdb.in‚0) disk./.usagePercent‚83.71018234343258$ disk./.usageGB‚2993.881591796875 disk.nvme0n1.in‚ 56.55859375 disk.rbd2.in‚0 disk.sda.in‚0 disk.rbd0.out‚0 disk.sdb.out‚ 95.0859375$ memory_percent‚98.64084720611572 cpu‚24.99944427860486 disk.sda1.out‚0 disk.dm-0.out‚ 8915.91015625 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 95.0859375 disk.sda2.out‚ 95.0859375" disk.nvme0n1.out‚ 3133.55859375* proc.memory.percent‚17.736101150512695 disk.md0.in‚0 disk.sdb2.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚0( proc.memory.availableMB‚ 222.68359375€wÊ0Ø+ _wandbruntime‚881  _runtime‚8810V> ÄÙ¾ & lr-AdamW/pg1‚0.0006954761245379516 & lr-AdamW/pg2‚0.0006954761245379516 trainer/global_step‚799 % _timestamp‚1.7884968232972896e+09 _runtime‚ 881.074649894 _step‚60ý×¼ÈÚ & lr-AdamW/pg2‚0.0006954761245379516 trainer/global_step‚799 % _timestamp‚1.7884968232972896e+09 _runtime‚ 881.074649894 _step‚60 & lr-AdamW/pg1‚0.0006954761245379516DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4684885 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 893668973424 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9659048 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16138792 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚73141425375744 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19908653 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚65319107458560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 897032275456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16138794 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16138794 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚66216139734016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚ 881.07595637 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚65319125461504 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16138792 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4684272 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚799 % _timestamp‚1.7884968232985208e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36047445 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚65319107458560 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19933499 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 897032275456 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 219063296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚73141182372352 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚64130504231264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1222 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚64129579224448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚66217066941440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14343320 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63235927290384 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14343935 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 H>DeviceStatsMonitor.on7ôÔÖy_train_batch_start/active.small_pool.peak‚30508 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 939628376064 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚615 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚72201553996288 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392*¸¼âÒ'ÝÌO YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 897032275456 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 219063296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚73141182372352 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚64130504231264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1222 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚64129579224448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚66217066941440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14343320 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63235927290384 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14343935 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 939628376064 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚615 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚72201553996288 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19908653 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚61 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 897941479936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 243003392 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 939847439360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 894576940880 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚66217066941440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9659050 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4684885 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 893668973424 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9659048 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16138792 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚73141425375744 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19908653 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚65319107458560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 897032275456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16138794 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16138794 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚66216139734016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚ 881.07595637 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚65319125461504 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16138792 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4684272 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚799 % _timestamp‚1.7884968232985208e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36047445 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚65319107458560 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19933499 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392=Œòé0Þ+ _wandbruntime‚882  _runtime‚882˜ã_#Ù&ßÓM E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16158954 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16158952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9671115 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚64210639099294 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19933484 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 895694214366 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14361248 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1222 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 940818359296 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63314944884928 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 894786246702 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19958356 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚63314927845568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚799 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 898153735680 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16158954 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19958356 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36117310 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4690759 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚65400726544384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚65400744547328 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚72291803640832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚72291779700736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚73232598060032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚626 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9671113 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚66298880280064 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚65400726544384 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 899062953472 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19933484 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 % _timestamp‚1.7884968243519356e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14361874 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44}À<Ý&à×M NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19958356 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36117310 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4690759 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚65400726544384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚65400744547328 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚72291803640832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚72291779700736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚73232598060032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚626 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9671113 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚66298880280064 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚65400726544384 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 899062953472 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19933484 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 % _timestamp‚1.7884968243519356e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14361874 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16158954 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16158952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9671115 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚64210639099294 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19933484 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 895694214366 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14361248 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1222 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 940818359296 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63314944884928 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 894786246702 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19958356 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚63314927845568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚799 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 898153735680 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16158954 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845>c¨²0á+ _wandbruntime‚882  _runtime‚882T _tTâÎ trainer/global_step‚799 /loss/FRt1/assignment_loss‚4.389896657906081 /loss/FRt3/assignment_loss‚3.897034382278269 /loss/FRt4/detection_loss‚2.2342587213190095 .loss/FBt4/detection_loss‚6.205053806304932 _runtime‚ 882.131017041 0loss/SRqqt1/detection_loss‚4.046834588050842 %loss/total_loss‚63.14725610814356 0loss/FBt3/assignment_loss‚0.8052505571395159 /loss/FRt3/detection_loss‚2.3014572113752365 /loss/FRt2/assignment_loss‚5.758448932481849 2loss/SRqqt4/assignment_loss‚2.3589208778880892 1loss/FBt1/assignment_loss‚0.29800680320631934 0loss/FBt2/assignment_loss‚0.7468142770897733 0loss/SRqqt4/detection_loss‚2.288567149922961 % _timestamp‚1.7884968243539796e+09 /loss/FRt2/detection_loss‚3.2705844848052315 2loss/SRqqt3/assignment_loss‚2.5277850085071156 .loss/FBt1/detection_loss‚0.664358981824059 epoch‚0 _step‚63 1loss/SRqqt3/detection_loss‚1.0424983702167603 .loss/FRt4/assignment_loss‚5.54504521387928 2loss/SRqqt1/assignment_loss‚1.7957389780453272 2loss/SRqqt2/assignment_loss‚2.5914977894109836 1loss/FBt4/assignment_loss‚0.18380138278007507 .loss/FBt2/detection_loss‚2.089397680428293 /loss/FBt3/detection_loss‚2.5598997846245766 /loss/FRt1/detection_loss‚2.2186550066659323 1loss/SRqqt2/detection_loss‚3.3274494619930493lgÍãÒ /loss/FRt1/detection_loss‚2.2186550066659323 1loss/SRqqt2/detection_loss‚3.3274494619930493 trainer/global_step‚799 /loss/FRt1/assignment_loss‚4.389896657906081 /loss/FRt3/assignment_loss‚3.897034382278269 /loss/FRt4/detection_loss‚2.234258721Ñ DWE3190095 .loss/FBt4/detection_loss‚6.205053806304932 _runtime‚ 882.131017041 0loss/SRqqt1/detection_loss‚4.046834588050842 %loss/total_loss‚63.14725610814356 0loss/FBt3/assignment_loss‚0.8052505571395159 /loss/FRt3/detection_loss‚2.3014572113752365 /loss/FRt2/assignment_loss‚5.758448932481849 2loss/SRqqt4/assignment_loss‚2.3589208778880892 1loss/FBt1/assignment_loss‚0.29800680320631934 0loss/FBt2/assignment_loss‚0.7468142770897733 0loss/SRqqt4/detection_loss‚2.288567149922961 % _timestamp‚1.7884968243539796e+09 /loss/FRt2/detection_loss‚3.2705844848052315 2loss/SRqqt3/assignment_loss‚2.5277850085071156 .loss/FBt1/detection_loss‚0.664358981824059 epoch‚0 _step‚63 1loss/SRqqt3/detection_loss‚1.0424983702167603 .loss/FRt4/assignment_loss‚5.54504521387928 2loss/SRqqt1/assignment_loss‚1.7957389780453272 2loss/SRqqt2/assignment_loss‚2.5914977894109836 1loss/FBt4/assignment_loss‚0.18380138278007507 .loss/FBt2/detection_loss‚2.089397680428293 /loss/FBt3/detection_loss‚2.5598997846245766?5n–“ä‚(:ˆ »—éÔò´¡m gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.379' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.59475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝÔ÷® å‚(:þ »—éÔ†èóm( proc.memory.availableMB‚ 222.60546875 cpu‚24.999988084250777 disk.sdb1.out‚0 disk.sda2.in‚0 disk.sda2.out‚ 96.99609375 disk.nvme0n1.in‚ 56.55859375 disk.sdb2.out‚ 96.99609375$ memory_percent‚98.64132404327393* proc.memory.percent‚17.736101150512695 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.sdb2.in‚0 network.sent‚ 1.091664e+06 disk.rbd0.in‚0 disk.dm-0.in‚ 62.390625$ disk./.usageGB‚2994.131076812744 disk.md0.in‚0 disk.sda.in‚0! disk.nvme0n1.out‚ 3134.5859375 disk.nvme1n1.in‚ 5.83203125" proc.memory.rssMB‚ 2905.8828125 disk.rbd0.out‚0 disk.sdb.in‚0 disk.sdb.out‚ 96.99609375 disk.md0.out‚ 93.28515625 network.recv‚369665 disk.sda.out‚ 96.99609375 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.out‚ 93.28515625 disk.dm-0.out‚ 9273.03125) disk./.usagePercent‚83.71715804889386 disk.md0p1.in‚0! disk.nvme1n1.out‚ 6138.4453125 disk.rbd2.in‚0 disk.rbd2.out‚0ûÒ‡D’æ‚(:‡ Ê—éÔωÄk gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚175.334' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.8335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿš6U ç‚(: Ê—éÔÈùñm disk.sda.out‚98.45751953125 network.recv‚374956 disk.nvme0n1.in‚ 56.55859375 disk.md0.out‚94.6875 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.md0p1.out‚94.6875 disk.nvme1n1.in‚ 5.83203125 disk.md0.in‚0* proc.memory.percent‚17.736101150512695 cpu‚25.00026029366034 disk.nvme0n1.out‚3134.625 disk.rbd2.out‚0 disk.sdb2.out‚98.45751953125 disk.rbd0.in‚0) disk./.usagePercent‚83.71715804889386 disk.sdb1.out‚0 disk.sda2.out‚98.45751953125 network.sent‚ 1.141053e+06$ disk./.usageGB‚2994.131076812744 disk.sdb1.in‚0 disk.sdb.out‚98.45751953125 disk.sda2.in‚0" disk.nvme1n1.out‚ 6395.06640625 disk.sda1.out‚0( proc.memory.availableMB‚ 222.60546875 disk.sdb.in‚0 disk.rbd2.in‚0 disk.sdb2.in‚0$ memory_percent‚98.64132404327393" proc.memory.rssMB‚ 2905.8828125 disk.md0p1.in‚0 disk.dm-0.in‚ 62.390625 disk.dm-0.out‚ 9529.69140625 disk.sda.in‚0ã[ê"“è‚(:ˆ Ù—éÔ¶ÀÙk gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚124.321' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.08025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØËšÝ é‚(: Ù—éÔ뚺l disk.md0.in‚0% disk./.usageGB‚2994.1009254455566 disk.sdb.in‚0 disk.sdb2.out‚ 99.9248046875 disk.dm-0.out‚ 9693.2265625* proc.memory.percent‚17.736101150512695 cpu‚25.000601118495545 disk.sda2.out‚ 99.9248046875 disk.nvme0n1.out‚3136.5 disk.rbd0.out‚0 disk.md0p1.out‚ 96.08984375 network.sent‚ 1.146307e+06 disk.sdb.out‚ 99.9248046875 disk.sda2.in‚0 disk.nvme0n1.in‚ 56.55859375 disk.sdb2.in‚0 network.recv‚378790 disk.rbd2.out‚0" proc.memory.rssMB‚ 2905.8828125 disk.md0.out‚ 96.08984375) disk./.usagePercent‚83.71631500404798 disk.nvme1n1.in‚ 5.83203125 proc.cpu.threads‚17! disk.nvme1n1.out‚ 6556.7265625 disk.rbd2.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚0 disk.dm-0.in‚ 62.390625 disk.sda.out‚ 99.9248046875$ memory_percent‚98.64301681518555& proc.memory.availableMB‚ 222.328125 disk.sda1.out‚0 disk.rbd0.in‚0Íwî[’ê‚(:‡ è—éÔ”´Ák gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚133.958' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.4895 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†c£“ë‚(:‹ è—éÔ⮳m! disk.nvme0n1.out‚ 3136.5546875 disk.sdb1.out‚0 network.sent‚ 1.152209e+06) disk./.usagePercent‚83.71631468406639 disk.sdb.out‚101.64501953125 proc.cpu.threads‚17! disk.sda2.out‚101.64501953125 disk.md0.out‚ 97.75390625 disk.sda1.in‚0* proc.memory.percent‚17.736101150512695 disk.nvme1n1.in‚ 5.83203125 disk.md0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.sda.in‚0 disk.sda.out‚101.64501953125 disk.rbd0.in‚0 disk.md0p1.in‚0 disk.nvme0n1.in‚ 56.55859375 disk.sdb2.in‚0 disk.sda1.out‚0 disk.sdb1.in‚0% proc.memory.availableMB‚ 222.34375" proc.memory.rssMB‚ 2905.8828125 network.recv‚383357$ disk./.usageGB‚2994.100914001465 disk.sda2.in‚0 disk.nvme1n1.out‚ 6668.890625 disk.dm-0.in‚ 62.390625 disk.dm-0.out‚ 9805.4453125 disk.sdb.in‚0 cpu‚25.001064312495753! disk.sdb2.out‚101.64501953125 disk.rbd0.out‚0 disk.md0p1.out‚ 97.75390625# memory_percent‚98.6429214477539‹ÅªÃ0ì+ _wandbruntime‚935  _runtime‚935b¸`üÄí¾ & lr-AdamW/pg1‚0.0007389977843963965 & lr-AdamW/pg2‚0.0007389977843963965 trainer/global_step‚849 % _timestamp‚1.7884968778231196e+09 _runtime‚ 935.600322685 _step‚64µ‚.@Èî & lr-AdamW/pg2‚0.0007389977843963965 trainer/global_step‚849 % _timestamp‚1.7884968778231196e+09 _runtime‚ 935.600322685 _step‚64 & lr-AdamW/pg1‚0.0007389977843963965@$Ø!x0ï+ _wandbruntime‚935  _runtime‚935˜UÑ'ðËO YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 953119700480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚849 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38300347 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 954028904960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 998727711744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17146792 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38325195 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚68137261213572 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚76712839218688 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚67186807017584 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚70353181450240 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚67186789978224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21178401 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 953119700480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15242461 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4980163 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚77711341575680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚70354108657664 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21178401 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17146792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 998502356992 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚68136336206756 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10262298 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70353181450240 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚68137261213572 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚76712839218688 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚67186807017584 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚70353181450240 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚67186789978224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21178401 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 953119700480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15242461 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4980163 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚77711341575680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚70354108657664 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21178401 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17146792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 998502356992 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚68136336206756 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10262298 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70353181450240 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38300347 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 954028904960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 998727711744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17146792 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38325195 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1225 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 950663501810 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 % _timestamp‚1.7884968788797107e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _step‚66 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚69481680835584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10274365 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 951571469474 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.smÆ8uÚall_pool.peak‚570 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚69481698838528 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚77803006495744 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 trainer/global_step‚849 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚69481680835584 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15260273 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4985910 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚69481698838528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38345338 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232504320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚67265824612128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17166954 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38345338 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17166952 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4986531 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 999917632512 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38370212 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚68217396081602 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208564224Š*}½ß&ôÙM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17166954 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38345338 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17166952 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4986531 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 999917632512 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38370212 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚68217396081602 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208564224 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1262 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚68216471074578 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21203258 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21178386 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21203258 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38370212 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1225 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 950663501810 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 % _timestamp‚1.7884968788797107e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _step‚66 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚69481680835584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10274365 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 951571469474 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚69481698838528 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚77803006495744 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 trainer/global_step‚849 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚69481680835584 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15260273 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4985910 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚69481698838528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38345338 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232504320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚67265824612128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784BBçƒL0õ+ _wandbruntime‚936  _runtime‚9363 ƒŒCö½ 2loss/SRqqt2/assignment_loss‚2.4698785468935966 2loss/SRqqt3/assignment_loss‚1.9431340988816284 2loss/SRqqt4/assignment_loss‚2.6223566432793937 1loss/SRqqt2/detection_loss‚3.5199293196201324 .loss/FBt2/detection_loss‚2.659197139112573 0loss/FBt2/assignment_loss‚0.6395642443655417 /loss/FRt1/detection_loss‚2.1673226050428442 /loss/FRt2/detection_loss‚3.1150162197925426 /loss/FRt4/detection_loss‚2.4953120296651665 %loss/total_loss‚57.65321576082838 1loss/SRqqt4/detection_loss‚2.7179189920425415 0loss/FBt3/assignment_loss‚0.9527756223615143 loss/FBt4/assignment_loss‚0 0loss/FBt1/assignment_loss‚0.3956108049348007 /loss/FRt3/detection_loss‚2.1945524094281375 1loss/SRqqt3/detection_loss‚0.9685580651995128 _runtime‚ 936.658893919 /loss/SRqqt1/detection_loss‚3.05947408080101 /loss/FBt3/detection_loss‚2.3417664766311646 epoch‚0 /loss/FRt1/assignment_loss‚3.697493201977498 .loss/FBt4/detection_loss‚2.087321400642395 % _timestamp‚1.7884968788816912e+09 /loss/FRt2/assignment_loss‚6.158264301441334 trainer/global_step‚849 /loss/FRt3/assignment_loss‚2.739011380407545 /loss/FRt4/assignment_loss‚5.294368456710469 2loss/SRqqt1/assignment_loss‚2.7918741766895567 _step‚67 /loss/FBt1/detection_loss‚0.6225155449074818sÙAÝG÷Á 1loss/SRqqt4/detection_loss‚2.7179189920425415 0loss/FBt3/assignment_loss‚0.9527756223615143 loss/FBt4/assignment_loss‚0 0loss/FBt1/assignment_loss‚0.3956108049348007 /loss/FRt3/detection_loss‚2.1945524094281375 1loss/SRqqt3/detection_loss‚0.9685580651995128 _runtime‚ 936.658893919 /loss/SRqqt1/detection_loss‚3.05947408080101 /loss/FBt3/detection_loss‚2.3417664766311646 epoch‚0 /loss/FRt1/assignment_loss‚3.697493201977498 .loss/FBt4/detection_loss‚2.087321400642395 % _timestamp‚1.7884968788816912e+09 /loss/FRt2/assignment_loss‚6.158264301441334 trainer/global_step‚849 /loss/FRt3/assignment_loss‚2.739011380407545 /loss/FRt4/assignment_loss‚5.294368456710469 2loss/SRqqt1/assignment_loss‚2.7918741766895567 _step‚67 /loss/FBt1/detection_loss‚0.6225155449074818 2loss/SRqqt2/assignment_loss‚2.4698785468935966 2loss/SRqqt3/assignment_loss‚1.9431340988816284 2loss/SRqqt4/assignment_loss‚2.6223566432793937 1loss/SRqqt2/detection_loss‚3.5199293196201324 .loss/FBt2/detection_loss‚2.659197139112573 0loss/FBt2/assignment_loss‚0.6395642443655417 /loss/FRt1/detection_loss‚2.1673226050428442 /loss/FRt2/detection_loss‚3.1150162197925426 /loss/FRt4/detection_loss‚2.4953120296651665 %loss/total_loss‚57.65321576082838CÉÙˆP‘ø‚(:† ÷—éÔÏÔ£l gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.344' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?(Dù‚(:’ ÷—éÔíýl disk.dm-0.in‚ 62.390625 disk.dm-0.out‚ 9904.1484375 disk.md0.out‚ 99.26953125 disk.sdb1.in‚0" disk.nvme1n1.out‚ 6767.42578125 disk.sda.in‚0 disk.md0p1.in‚0 disk.sdb2.in‚0$ memory_percent‚98.64180088043213% disk./.usageGB‚2994.0865898132324 disk.md0p1.out‚ 99.26953125" disk.nvme0n1.out‚ 3136.72265625 disk.rbd2.out‚0* proc.memory.percent‚17.736101150512695 cpu‚25.00158505785203 network.sent‚ 1.157956e+06 disk.sda2.in‚0 disk.nvme0n1.in‚ 56.55859375 disk.nvme1n1.in‚ 5.83203125 disk.md0.in‚0 disk.sda.out‚103.22607421875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0( proc.memory.availableMB‚ 222.52734375 proc.cpu.threads‚17 network.recv‚387272 disk.sdb.in‚0 disk.sdb.out‚103.22607421875! disk.sda2.out‚103.22607421875! disk.sdb2.out‚103.22607421875" proc.memory.rssMB‚ 2905.8828125) disk./.usagePercent‚83.71591417376827 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda1.out‚0ïèÑu“ú‚(:ˆ †˜éÔŒä–m gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚178.297' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.57425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄÉOû‚(:Œ †˜éÔ˜–Øn disk.nvme1n1.in‚ 5.83203125 disk.rbd2.out‚0 disk.md0p1.in‚0( disk./.usagePercent‚83.7172156455812 disk.sda2.out‚104.4814453125 disk.nvme0n1.in‚ 56.55859375" disk.nvme0n1.out‚ 3223.00390625 disk.md0.in‚0 disk.sda1.in‚0 disk.dm-0.out‚ 10147.7734375 disk.sdb.in‚0$ memory_percent‚98.64296913146973* proc.memory.percent‚17.736101150512695" disk.nvme1n1.out‚ 6924.76953125 disk.md0.out‚ 100.4609375 disk.sdb2.in‚0 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 62.390625 network.recv‚393113% disk./.usageGB‚2994.1331367492676 disk.sdb2.out‚104.4814453125 disk.sda1.out‚0' proc.memory.availableMB‚ 222.3359375 cpu‚25.0017244498657 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd2.in‚0 disk.sda.out‚104.4814453125 disk.md0p1.out‚ 100.4609375" proc.memory.rssMB‚ 2905.8828125 network.sent‚ 1.207233e+06 disk.sdb.out‚104.4814453125 disk.sda.in‚0 disk.rbd0.out‚0^w=“ü‚(:ˆ •˜éÔ¡û½k gpu.0.gpu‚16.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚117.525' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.38125 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:·ý‚(:“ •˜éÔ¾ª²l% disk./.usageGB‚2994.1955223083496 disk.sdb.out‚106.76708984375" disk.nvme1n1.out‚ 7041.38671875 disk.rbd2.in‚0 disk.rbd0.out‚0 network.recv‚396380 disk.rbd2.out‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 2905.8828125 disk.md0p1.out‚ 102.6796875 disk.dm-0.in‚ 62.390625 disk.md0.in‚0 cpu‚25.002317616849172 disk.md0p1.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0$ memory_percent‚98.64375591278076) disk./.usagePercent‚83.71895997192354 disk.dm-0.out‚ 10300.140625 disk.nvme1n1.in‚ 5.83203125! disk.sdb2.out‚106.76708984375* proc.memory.percent‚17.736101150512695 disk.nvme0n1.in‚ 56.55859375" disk.nvme0n1.out‚ 3258.75390625 disk.md0.out‚ 102.6796875 disk.sdb2.in‚0 disk.sda.out‚106.76708984375( proc.memory.availableMB‚ 222.20703125! disk.sda2.out‚106.76708984375 disk.sdb.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 network.sent‚ 1.212411e+06«d„0þ+ _wandbruntime‚989  _runtime‚989ø3φÃÿ½ $ _timestamp‚1.788496931832269e+09 _runtime‚ 989.609455387 _step‚68 & lr-AdamW/pg1‚0.0007825194442548416 & lr-AdamW/pg2‚0.0007825194442548416 trainer/global_step‚899ÂñK!Ç€Á & lr-AdamW/pg1‚0.0007825194442548416 & lr-AdamW/pg2‚0.0007825194442548416 trainer/global_step‚899 $ _timestamp‚1.788496931832269e+09 _runtime‚ 989.609455387 _step‚68DƒÁ¨0+ _wandbruntime‚989  _runtime‚989#N.1*'‚ÓO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10865550 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10865548 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18154792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884969318329709e+09 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚72143093189064 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚71137686744784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1225 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚73481034043904 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚82282098692096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚72144018195880 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74490223166464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1262 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1009207125504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚74490223166464 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22398457 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40578097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16142827 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚74491150373888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18154792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚73481016040960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1057737539072 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5276637 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236711936 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1057950310912 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18154794 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18154794 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚(2n¯81224124441088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0öÕͲÝ'ƒ×O M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40578097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16142827 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚74491150373888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18154792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚73481016040960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1057737539072 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5276637 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236711936 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1057950310912 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18154794 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18154794 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚81224124441088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10865550 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10865548 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18154792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884969318329709e+09 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚72143093189064 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚71137686744784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1225 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚73481034043904 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚82282098692096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚72144018195880 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74490223166464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1262 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1009207125504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚74490223166464 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22398457E³ËBþ“„‚(:ˆ ¤˜éÔ¯¬²m gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚178.449' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.61225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à“K…‚(:À ¤˜éÔòÓóm disk.sdb2.out‚ 108.283203125 disk.rbd0.out‚0 disk.md0p1.out‚ 104.14453125 disk.md0.out‚ 104.14453125* proc.memory.percent‚17.736101150512695 proc.cpu.threads‚17! disk.nvme1n1.out‚ 7082.7421875 network.recv‚400767 disk.sdb2.in‚ 0.23046875 disk.sda.out‚ 108.283203125 disk.rbd0.in‚0" disk.nvme0n1.out‚ 3356.41796875 disk.md0.in‚ 1.62890625" proc.memory.rssMB‚ 2905.8828125) disk./.usagePercent‚83.69474120485785% disk./.usageGB‚2993.3293418884277 disk.sdb1.in‚0 disk.sda2.in‚ 1.3984375 disk.nvme0n1.in‚ 56.55859375$ memory_percent‚98.64530563354492 cpu‚25.002451668260953 network.sent‚ 1.218325e+06 disk.rbd2.out‚0 disk.sdb.out‚ 108.283203125 disk.rbd2.in‚0 disk.sda1.in‚0 disk.nvme1n1.in‚ 5.83203125 disk.md0p1.in‚ 1.62890625 disk.dm-0.in‚ 62.390625 disk.dm-0.out‚10439.16015625& proc.memory.availableMB‚ 221.953125 disk.sdb1.out‚0 disk.sdb.in‚ 0.23046875 disk.sda2.out‚ 108.283203125 disk.sda.in‚ 1.3984375 disk.sda1.out‚0‡Ø|¶0†+ _wandbruntime‚990  _runtime‚990G“Ýã&‡ÝM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚82373268908544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1059139910144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40623114 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 _runtime‚ 990.564380435 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚73562635126784 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10877613 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚74573890933248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18174952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40598240 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40598240 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22423288 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22448160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 % _timestamp‚1.7884969327865443e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5283149 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚81314350145536 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1007448724582 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚73562653129728 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10877615 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚634 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _step‚70 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1010328585728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18174952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16160764 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1011237803520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚71216704339328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚72223228056886 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚81314374085632 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18174954 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1262 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1058918763008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5282515 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16160128 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82373513995776 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1011237803520 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1225 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10877615 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚634 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _step‚70 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1010328585728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18174952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16160764 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1011237803520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚71216704339328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚72223228056886 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚81314374085632 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18174954 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1262 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1058918763008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5282515 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16160128 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82373513995776 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1011237803520 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1225 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40623114 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 _runtime‚ 990.564380435 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚73562635126784 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10877613 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚74573890933248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18174952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40598240 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40598240 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22423288 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22448160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 % _timestamp‚1.7884969327865443e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5283149 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚81314350145536 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1007448724582 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845FW¹HŽ0‰+ _wandbruntime‚990  _runtime‚990ûB-¢-ŠÎ /loss/FRt4/detection_loss‚2.6532006191044317 1loss/SRqqt3/detection_loss‚0.8920995205759056 &loss/total_loss‚60.829582049674336 0loss/FBt3/assignment_loss‚0.7110660374164571 .loss/FBt4/detection_loss‚2.913968230996813 epoch‚0 trainer/global_step‚899 % _timestamp‚1.7884969327883718e+09 /loss/FBt1/assignment_loss‚0.250796400436062 /loss/FRt4/assignment_loss‚5.681002628512498 2loss/SRqqt1/assignment_loss‚2.9265509943167367 .loss/FRt1/detection_loss‚2.090384225361049 .loss/FBt2/detection_loss‚4.511933986097574 0loss/FBt2/assignment_loss‚0.5219907294957551 .loss/FRt3/detection_loss‚1.783451253438697 1loss/SRqqt1/detection_loss‚2.7508702824513116 /loss/FBt1/detection_loss‚0.7785164463056925 _step‚71 2loss/SRqqt4/assignment_los¡˜º's‚2.7316728063992093 0loss/SRqqt2/detection_loss‚2.961136889954408 1loss/SRqqt4/detection_loss‚2.5371073348181588 .loss/FBt3/detection_loss‚4.151336201599666 0loss/FRt1/assignment_loss‚3.1673572156578302 /loss/FRt3/assignment_loss‚3.085408299284823 2loss/SRqqt3/assignment_loss‚2.0341089184558583 0loss/FBt4/assignment_loss‚0.1651679820248059 _runtime‚ 990.565336812 0loss/FRt2/assignment_loss‚5.5722463381917855 2loss/SRqqt2/assignment_loss‚2.5085914358496666 /loss/FRt2/detection_loss‚3.4496172729291414çþš1X‹Ò 2loss/SRqqt4/assignment_loss‚2.7316728063992093 0loss/SRqqt2/detection_loss‚2.961136889954408 1loss/SRqqt4/detection_loss‚2.5371073348181588 .loss/FBt3/detection_loss‚4.151336201599666 0loss/FRt1/assignment_loss‚3.1673572156578302 /loss/FRt3/assignment_loss‚3.085408299284823 2loss/SRqqt3/assignment_loss‚2.0341089184558583 0loss/FBt4/assignment_loss‚0.1651679820248059 _runtime‚ 990.565336812 0loss/FRt2/assignment_loss‚5.5722463381917855 2loss/SRqqt2/assignment_loss‚2.5085914358496666 /loss/FRt2/detection_loss‚3.4496172729291414 /loss/FRt4/detection_loss‚2.6532006191044317 1loss/SRqqt3/detection_loss‚0.8920995205759056 &loss/total_loss‚60.829582049674336 0loss/FBt3/assignment_loss‚0.7110660374164571 .loss/FBt4/detection_loss‚2.913968230996813 epoch‚0 trainer/global_step‚899 % _timestamp‚1.7884969327883718e+09 /loss/FBt1/assignment_loss‚0.250796400436062 /loss/FRt4/assignment_loss‚5.681002628512498 2loss/SRqqt1/assignment_loss‚2.9265509943167367 .loss/FRt1/detection_loss‚2.090384225361049 .loss/FBt2/detection_loss‚4.511933986097574 0loss/FBt2/assignment_loss‚0.5219907294957551 .loss/FRt3/detection_loss‚1.783451253438697 1loss/SRqqt1/detection_loss‚2.7508702824513116 /loss/FBt1/detection_loss‚0.7785164463056925 _step‚71G®ç ª’Œ‚(:‡ ³˜éÔ†©l gpu.0.gpu‚9.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹–ˆA‚(:¶ ³˜éÔ³Ül disk.dm-0.in‚ 62.390625 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sda.out‚ 110.103515625 disk.rbd0.in‚0 disk.rbd0.out‚0 cpu‚25.002961491332144 network.sent‚ 1.255931e+06! disk.nvme0n1.out‚ 3452.5859375 disk.sdb1.in‚0$ memory_percent‚98.64435195922852& proc.memory.availableMB‚ 222.109375 network.recv‚405530 disk.dm-0.out‚ 10623.78125 disk.nvme0n1.in‚ 56.55859375! disk.nvme1n1.out‚ 7171.1953125 disk.md0.out‚ 105.90625 disk.sda1.in‚0" proc.memory.rssMB‚ 2905.8828125) disk./.usagePercent‚83.70174880181824 disk.sda2.out‚ 110.103515625% disk./.usageGB‚2993.5799674987793 disk.sdb.out‚ 110.103515625 disk.sdb2.in‚ 0.23046875 disk.md0p1.in‚ 1.62890625 disk.nvme1n1.in‚ 5.83203125 disk.md0.in‚ 1.62890625 disk.rbd2.out‚0 disk.sdb2.out‚ 110.103515625 disk.md0p1.out‚ 105.90625* proc.memory.percent‚17.736101150512695 disk.sdb.in‚ 0.23046875 disk.sda2.in‚ 1.3984375 disk.sda.in‚ 1.3984375 disk.sda1.out‚0VNQ¨“Ž‚(:ˆ ˜éÔËÅm gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚159.079' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.76975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X‹-Q‚(:Æ ˜éÔâân disk.rbd2.out‚0 disk.sda1.in‚0 disk.md0p1.out‚ 107.26953125 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sdb1.out‚0! disk.sda2.out‚111.52392578125! disk.nvme0n1.out‚ 3551.8359375" disk.nvme1n1.out‚ 7171.33984375 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-0.out‚10723.17578125 disk.md0p1.in‚ 1.62890625 disk.sdb.in‚ 0.23046875 disk.nvme0n1.in‚ 56.55859375 disk.nvme1n1.in‚ 5.83203125 disk.rbd2.in‚0 cpu‚25.00305660636685 disk.sda.in‚ 1.3984375 disk.sda.out‚111.52392578125 disk.dm-0.in‚ 62.390625* proc.memory.percent‚17.736101150512695 disk.sdb1.in‚0" proc.memory.rssMB‚ 2905.8828125) disk./.usagePercent‚83.70171883020872% disk./.usageGB‚2993.5788955688477 disk.sda2.in‚ 1.3984375 disk.md0.out‚ 107.26953125! disk.sdb2.out‚111.52392578125 network.sent‚ 1.297198e+06 network.recv‚410997 disk.sdb.out‚111.52392578125 disk.md0.in‚ 1.62890625 disk.sdb2.in‚ 0.23046875$ memory_percent‚98.64253997802734% proc.memory.availableMB‚ 222.40625£Â&Q‘‚(:† јéÔœ°îl gpu.0.gpu‚11.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚182.456' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.614 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾b±ßB‘‚(:· јéÔлm disk.sdb1.out‚0 disk.md0p1.in‚ 1.62890625 disk.sda2.in‚ 1.3984375 disk.nvme0n1.out‚ 3682.15625 disk.md0.in‚ 1.62890625 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 62.390625 disk.sdb.out‚ 113.6953125 disk.sda.in‚ 1.3984375 disk.rbd0.out‚0' proc.memory.availableMB‚ 222.1015625" proc.memory.rssMB‚ 2905.8828125) disk./.usagePercent‚83.69918532258923 disk.sda.out‚ 113.6953125 disk.sda1.out‚0 disk.nvme0n1.in‚ 56.55859375" disk.nvme1n1.out‚ 7172.59765625 disk.sdb2.in‚ 0.23046875 disk.sdb2.out‚ 113.6953125 disk.sda1.in‚0$ memory_percent‚98.64439964294434 disk.sdb1.in‚0 disk.md0p1.out‚ 109.37890625 disk.sdb.in‚ 0.23046875% disk./.usageGB‚2993.4882850646973 disk.dm-0.out‚10854.75390625 disk.nvme1n1.in‚ 5.83203125 disk.md0.out‚ 109.37890625 disk.sda2.out‚ 113.6953125 proc.cpu.threads‚17 network.sent‚ 1.300362e+06 network.recv‚414414* proc.memory.percent‚17.736101150512695 cpu‚25.003312530212686,ô³2’- _wandbruntime‚1040  _runtime‚10401»Å“¿ _step‚72 & lr-AdamW/pg1‚0.0008260411041132866 & lr-AdamW/pg2‚0.0008260411041132866 trainer/global_step‚949 % _timestamp‚1.7884969831194317e+09 _runtime‚1040.896744422í¹ªÉ”Ã % _timestamp‚1.7884969831194317e+09 _runtime‚1040.896744422 _step‚72 & lr-AdamW/pg1‚0.0008260411041132866 & lr-AdamW/pg2‚0.0008260411041132866 trainer/global_step‚949H×5Ù›2•- _wandbruntime‚1040  _runtime‚10402&æÀÚ'–ÔO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚663 % _timestamp‚1.7884969831204798e+09 _step‚73 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1065294550528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚76150775178188 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1066203755008 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1262 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚78627264882688 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23668205 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚77561988335104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23668205 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1116677530624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚76149850171372 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5572927 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5573588 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚75088566471984 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚77561988335104 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23643359 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚75088549432624 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42830999 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 219063296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85735409663488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚78628192090112 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19162794 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42806151 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚78627264882688 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚77561970332160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚661 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚1040.897852036 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1066203755008 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86852087194112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1062208706204 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 trainer/global_step‚949 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚78628192090112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1061300738748 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19162792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17041725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11468798 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1225 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 243003392 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85735433603584 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5572927 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5573588 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚75088566471984 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚77561988335104 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23643359 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚75088549432624 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42830999 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 219063296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85735409663488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚78628192090112 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19162794 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42806151 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚78627264882688 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚77561970332160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚661 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚1040.897852036 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1066203755008 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86852087194112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1062208706204 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 trainer/global_step‚949 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚78628192090112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1061300738748 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19162792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17041725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11468798 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1225 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 243003392 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85735433603584 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚76150775178188 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1066203755008 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1262 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚78627264882688 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23668205 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚77561988335104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23668205 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1116677530624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0I…Êõ÷2˜- _wandbruntime‚1041  _runtime‚1041ƒ‡|¤«™ÞM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1262 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85825635367936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23693062 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17059557 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚76230910046218 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚74 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19182952 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1066416010752 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42851142 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 [HDeviceStatsMonitor.on_train_batch_end/inactive_sp ý‰Ì9lit_bytes.all.allocated‚86943739535872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚78710932649472 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚77643589417984 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42876016 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23668190 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17060208 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1117859080704 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1067325228544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚75167567027168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19182954 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85825659308032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23668190 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚78710005428736 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚86943494448640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚651 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11480865 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1062418012026 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1118080227840 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23693062 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚649 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚75167584066528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640’ظè&šâM VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚78710932649472 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚77643589417984 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42876016 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23668190 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17060208 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1117859080704 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1067325228544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚75167567027168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19182954 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85825659308032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23668190 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚78710005428736 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚86943494448640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚651 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11480865 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1062418012026 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1118080227840 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23693062 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚649 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚75167584066528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1262 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85825635367936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23693062 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17059557 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚76230910046218 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚74 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19182952 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1066416010752 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42851142 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚86943739535872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216Jñ½l½2›- _wandbruntime‚1041  _runtime‚1041 CjVœÐ % _timestamp‚1.7884969840856724e+09 /loss/FRt1/assignment_loss‚3.499493737305914 /loss/FRt2/assignment_loss‚5.850974520047505 1loss/SRqqt2/assignment_loss‚2.358055504099015 .loss/FRt3/detection_loss‚1.954574428498745 1loss/SRqqt3/detection_loss‚0.9985212331037905 trainer/global_step‚949 _step‚75 2loss/SRqqt1/assignment_loss‚3.0355746299028397 0loss/FBt2/assignment_loss‚0.6318542794746339 0loss/SRqqt4/detection_loss‚2.790895957013835 _runtime‚1041.862767745 2loss/SRqqt3/assignment_loss‚1.9696671311759342 1loss/FBt1/assignment_loss‚0.23325885232389862 1loss/SRqqt1/detection_loss‚3.3323230147361755 1loss/SRqqt2/detection_loss‚2.7242589958252443 epoch‚0 /loss/FRt4/assignment_loss‚5.863759542122866 2loss/SRqqt4/assignment_loss‚2.3589264076689016 0loss/FBt4/assignment_loss‚0.8216085806488991 .loss/FBt2/detection_loss‚2.739711658521132 /loss/FRt1/detection_loss‚2.0139165510024344 /loss/FBt1/detection_loss‚0.7524597531158066 /loss/FBt3/detection_loss‚2.8302481174468994 .loss/FBt4/detection_loss‚5.273727814356486 0loss/FRt3/assignment_loss‚3.0137472841888666 /loss/FRt2/detection_loss‚3.3094648818174996 /loss/FRt4/detection_loss‚2.7017006935217442 %loss/total_loss‚61.79108722706252 0loss/FBt3/assignment_loss‚0.7323636591434479òÜ=RZÔ % _timestamp‚1.7884969840856724e+09 /loss/FRt1/assignment_loss‚3.499493737305914 /loss/FRt2/assignment_loss‚5.850974520047505 1loss/SRqqt2/assignment_loss‚2.358055504099015 .loss/FRt3/detection_loss‚1.954574428498745 1loss/SRqqt3/detection_loss‚0.9985212331037905 trainer/global_step‚949 _step‚75 2loss/SRqqt1/assignment_loss‚3.0355746299028397 0loss/FBt2/assignment_loss‚0.6318542794746339 0loss/SRqqt4/detection_loss‚2.790895957013835 _runtime‚1041.862767745 2loss/SRqqt3/assignment_loss‚1.9696671311759342 1loss/FBt1/assignment_loss‚0.23325885232389862 1loss/SRqqt1/detection_loss‚3.3323230147361755 1loss/SRqqt2/detection_loss‚2.7242589958252443 epoch‚0 /loss/FRt4/assignment_loss‚5.863759542122866 2loss/SRqqt4/assignment_loss‚2.3589264076689016 0loss/FBt4/assignment_loss‚0.8216085806488991 .loss/FBt2/detection_loss‚2.739711658521132 /loss/FRt1/detection_loss‚2.0139165510024344 /loss/FBt1/detection_loss‚0.7524597531158066 /loss/FBt3/detection_loss‚2.8302481174468994 .loss/FBt4/detection_loss‚5.273727814356486 0loss/FRt3/assignment_loss‚3.0137472841888666 /loss/FRt2/detection_loss‚3.3094648818174996 /loss/FRt4/detection_loss‚2.7017006935217442 %loss/total_loss‚61.79108722706252 0loss/FBt3/assignment_loss‚0.7323636591434479K‡^¦“ž‚(:ˆ à˜éԴˆm gpu.0.gpu‚17.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.715' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.17875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hzLŸ‚(:Á à˜éÔнn disk.rbd2.in‚0 disk.rbd2.out‚0 disk.sdb2.in‚ 0.23046875( proc.memory.availableMB‚ 222.00390625* proc.memory.percent‚17.736101150512695 cpu‚25.003472386096913 disk.sdb1.out‚0! disk.nvme0n1.out‚ 3747.2734375 proc.cpu.threads‚17! disk.sdb2.out‚115.18701171875) disk./.usagePercent‚83.70312280279305$ disk./.usageGB‚2993.629108428955 disk.rbd0.in‚0 disk.md0p1.out‚110.8125 disk.dm-0.in‚ 62.390625 disk.sdb.out‚115.18701171875 disk.sda.in‚ 1.3984375 disk.rbd0.out‚0 disk.md0p1.in‚ 1.62890625 disk.dm-0.out‚10991.96484375 disk.sdb.in‚ 0.23046875 disk.sda2.in‚ 1.3984375 disk.nvme0n1.in‚ 56.55859375 network.sent‚ 1.306374e+06 network.recv‚419185 disk.md0.in‚ 1.62890625 disk.sda.out‚115.18701171875" proc.memory.rssMB‚ 2905.8828125 disk.nvme1n1.in‚ 5.83203125 disk.sda1.in‚0! disk.sda2.out‚115.18701171875$ memory_percent‚98.64499568939209 disk.sda1.out‚0 disk.sdb1.in‚0" disk.nvme1n1.out‚ 7244.69140625 disk.md0.out‚110.8125 œ¸)’ ‚(:‡ ï˜éÔ¶ ël gpu.0.gpu‚51.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚174.562' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.6405 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºa ñC¡‚(:¸ ï˜éÔƒê­m disk.sda2.in‚ 1.3984375 disk.md0p1.in‚ 1.62890625 disk.dm-0.out‚ 11162.0078125 disk.rbd2.in‚0 disk.sdb2.out‚ 116.740234375 disk.rbd0.out‚0 disk.md0.in‚ 1.62890625 disk.sdb2.in‚ 0.23046875 disk.sda.out‚ 116.740234375 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.out‚112.3125' proc.memory.availableMB‚ 221.8046875* proc.memory.percent‚17.736101150512695 proc.cpu.threads‚17 network.recv‚424089 disk.sdb1.out‚0 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.dm-0.in‚ 62.390625 disk.sdb.out‚ 116.740234375 disk.nvme0n1.in‚ 56.55859375 disk.nvme1n1.in‚ 5.83203125 disk.rbd2.out‚0" proc.memory.rssMB‚ 2905.8828125 network.sent‚ 1.355299e+06" disk.nvme0n1.out‚ 3845.12109375" disk.nvme1n1.out‚ 7316.88671875 disk.sda.in‚ 1.3984375) disk./.usagePercent‚83.70323500967282% disk./.usageGB‚2993.6331214904785 disk.md0.out‚112.3125$ memory_percent‚98.64621162414551 disk.sdb.in‚ 0.23046875 disk.sda2.out‚ 116.740234375 cpu‚25.00391432501195FòÃõ‘¢‚(:† þ˜éÔŸËÌm gpu.0.gpu‚37.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.916' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.229 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"ÃÖÏc£‚(:Ø þ˜éÔëãn& proc.memory.availableMB‚ 222.296875 disk.rbd0.in‚0 network.recv‚429040 disk.sda.out‚118.57763671875 disk.sda1.in‚0! disk.sda2.out‚118.57763671875 disk.rbd2.in‚0 disk.rbd2.out‚0.078125 disk.md0p1.in‚ 1.62890625 disk.md0.out‚ 114.08203125 disk.rbd0.out‚ 0.01171875% disk./.usageGB‚2993.1654090881348 disk.sda.in‚ 1.3984375 disk.md0p1.out‚ 114.08203125 disk.sdb.out‚118.57763671875" proc.memory.rssMB‚ 2905.8828125! disk.nvme1n1.out‚ 7373.8046875 disk.sdb1.out‚0! disk.sdb2.out‚118.57763671875 disk.sda1.out‚0 disk.sdb.in‚ 0.23046875 cpu‚25.004020634440092 network.sent‚ 1.363713e+06 disk.dm-0.in‚ 62.390625 disk.sda2.in‚ 1.3984375 disk.nvme0n1.in‚ 56.55859375" disk.nvme0n1.out‚ 3900.13671875 disk.nvme1n1.in‚ 5.83203125$ memory_percent‚98.64320755004883* proc.memory.percent‚17.736101150512695 disk.sdb1.in‚0 disk.sdb2.in‚ 0.23046875 proc.cpu.threads‚17) disk./.usagePercent‚83.69015757515045 disk.dm-0.out‚11273.94140625 disk.md0.in‚ 1.62890625\§Ï22¤- _wandbruntime‚1094  _runtime‚1094ˆ™sÅ¥¿ & lr-AdamW/pg1‚0.0008695627639717316 & lr-AdamW/pg2‚0.0008695627639717316 trainer/global_step‚999 % _timestamp‚1.7884970368196273e+09 _runtime‚1094.596855257 _step‚76$XŠRɦà & lr-AdamW/pg1‚0.0008695627639717316 & lr-AdamW/pg2‚0.0008695627639717316 trainer/global_step‚999 % _timestamp‚1.7884970368196273e+09 _runtime‚1094.596855257 _step‚76L(ÐVx2§- _wandbruntime‚1094  _runtime‚1094Ì}Ú'¨ÔO XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚81642924623360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45059053 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1121381975552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17941266 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12072050 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24913107 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚82765233806336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20170792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24888261 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚653 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1175900916224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1117177993856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚81642942626304 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17940611 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20170792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 trainer/global_step‚999 % _timestamp‚1.7884970368205929e+09 _step‚77 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82764306598912 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80156607153680 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20170794 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12072048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1175688144384 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24913107 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236711936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1226 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚79039429159824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1122291180032 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚82764306598912 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1263 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5868563 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212771840 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚81642924623360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚91422619742208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _runtime‚1094.597923042 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚81642942626304 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1122291180032 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360|Fgb©ØO F@DeviceSt¢ó+éÌ'atsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚91422619742208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _runtime‚1094.597923042 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚81642942626304 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1122291180032 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚81642924623360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45059053 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1121381975552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17941266 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12072050 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24913107 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚82765233806336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20170792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24888261 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚653 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1175900916224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1117177993856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚81642942626304 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17940611 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20170792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 trainer/global_step‚999 % _timestamp‚1.7884970368205929e+09 _step‚77 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82764306598912 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80156607153680 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20170794 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12072048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1175688144384 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24913107 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236711936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1226 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚79039429159824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1122291180032 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚82764306598912 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1263 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5868563 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212771840 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚81642924623360Mq,*㑪‚(:† ™éÔéŠál gpu.0.gpu‚52.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚135.184' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p‚OH«‚(:½ ™éÔ±êŒm disk.nvme0n1.in‚ 56.55859375 disk.nvme1n1.in‚ 5.83203125$ memory_percent‚98.64535331726074 disk.rbd2.out‚0.078125 disk.sdb.out‚ 120.15625 disk.sda2.out‚ 120.15625 disk.md0p1.out‚ 115.59765625 disk.md0.out‚ 115.59765625 disk.rbd0.in‚0 disk.sdb1.out‚0" proc.memory.rssMB‚ 2905.8828125 disk.sdb1.in‚0 disk.dm-0.in‚ 62.390625" disk.nvme1n1.out‚ 7426.41015625 disk.md0p1.in‚ 1.62890625 disk.sdb.in‚ 0.23046875 network.sent‚ 1.366596e+06) disk./.usagePercent‚83.69543428499557 disk.sdb2.in‚ 0.23046875 disk.sdb2.out‚ 120.15625 disk.rbd0.out‚ 0.07421875 proc.cpu.threads‚17" disk.nvme0n1.out‚ 3900.40234375 network.recv‚431789 disk.sda.in‚ 1.3984375 disk.sda.out‚ 120.15625 disk.sda1.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 221.9453125 cpu‚25.00449104973762 disk.md0.in‚ 1.62890625 disk.sda2.in‚ 1.3984375* proc.memory.percent‚17.736101150512695% disk./.usageGB‚2993.3541297912598 disk.rbd2.in‚0 disk.dm-0.out‚ 11326.8125’ÁH2¬- _wandbruntime‚1095  _runtime‚1095lH!Âä&­ÞM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240892928 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚91513793233408 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1122503435776 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45104044 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1177089595904 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216952832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚644 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚81724543709184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚82847047144960 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20190952 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12084113 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884970378823354e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1176872643072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45128918 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1263 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12084115 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚642 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚999 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45128918 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1122503435776 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20190954 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24913092 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80237667028526 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚79118446754368 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17958473 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚81724561712128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚81724561712128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 _step‚78 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1123412653568 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5874360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚90336944530432 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640$ ~‹è&®âM EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17958473 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚81724561712128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚81724561712128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 _step‚78 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1123412653568 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5874360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚90336944530432 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240892928 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚91513793233408 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1122503435776 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45104044 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1177089595904 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216952832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚644 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚81724543709184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚82847047144960 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20190952 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12084113 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884970378823354e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1176872643072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45128918 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1263 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12084115 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚642 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚999 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45128918 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1122503435776 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20190954 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24913092 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80237667028526 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚79118446754368Næ¶ÑË2¯- _wandbruntime‚1095  _runtime‚10952½Hgó°Ò /loss/FBt1/detection_loss‚0.7007454518360948 0loss/FRt1/assignment_loss‚3.6114128984903036 2loss/SRqqt1/assignment_loss‚2.6489086197316647 1loss/SRqqt3/assignment_loss‚1.805975740957529 1loss/SRqqt4/assignment_loss‚2.367178209871054 /loss/FBt4/detection_loss‚3.3773252591490746 &loss/total_loss‚55.780742492612376 0loss/FBt2/assignment_loss‚0.5590076552970064 1loss/SRqqt4/detection_loss‚2.4646098524332047 % _timestamp‚1.7884970378842022e+09 0loss/FRt3/assignment_loss‚2.2932840396057474 /loss/FRt4/assignment_loss‚5.631649227703319 1loss/SRqqt3/detection_loss‚0.8486479583259364 .loss/FBt2/detection_loss‚2.597690890942301 _step‚79 2loss/SRqqt2/assignment_loss‚1.9968095693205083 0loss/FBt3/assignment_loss‚0.3207229930907488 /loss/FRt3/detection_loss‚1.7165804789824919 /loss/FRt4/detection_loss‚2.8312886707922993 1loss/SRqqt2/detection_loss‚2.4680123180150986 .loss/FBt3/detection_loss‚2.781774652004242 trainer/fò©Eeglobal_step‚999 1loss/FBt1/assignment_loss‚0.24461937277314502 1loss/FBt4/assignment_loss‚0.27988380983879324 _runtime‚1095.661232553 /loss/FRt2/assignment_loss‚5.313067828907686 .loss/FRt1/detection_loss‚2.492027085274458 .loss/FRt2/detection_loss‚3.746381598360398 1loss/SRqqt1/detection_loss‚2.6831383109092712 epoch‚0ÌÌûk\±Ö 0loss/FRt1/assignment_loss‚3.6114128984903036 2loss/SRqqt1/assignment_loss‚2.6489086197316647 1loss/SRqqt3/assignment_loss‚1.805975740957529 1loss/SRqqt4/assignment_loss‚2.367178209871054 /loss/FBt4/detection_loss‚3.3773252591490746 &loss/total_loss‚55.780742492612376 0loss/FBt2/assignment_loss‚0.5590076552970064 1loss/SRqqt4/detection_loss‚2.4646098524332047 % _timestamp‚1.7884970378842022e+09 0loss/FRt3/assignment_loss‚2.2932840396057474 /loss/FRt4/assignment_loss‚5.631649227703319 1loss/SRqqt3/detection_loss‚0.8486479583259364 .loss/FBt2/detection_loss‚2.597690890942301 _step‚79 2loss/SRqqt2/assignment_loss‚1.9968095693205083 0loss/FBt3/assignment_loss‚0.3207229930907488 /loss/FRt3/detection_loss‚1.7165804789824919 /loss/FRt4/detection_loss‚2.8312886707922993 1loss/SRqqt2/detection_loss‚2.4680123180150986 .loss/FBt3/detection_loss‚2.781774652004242 trainer/global_step‚999 1loss/FBt1/assignment_loss‚0.24461937277314502 1loss/FBt4/assignment_loss‚0.27988380983879324 _runtime‚1095.661232553 /loss/FRt2/assignment_loss‚5.313067828907686 .loss/FRt1/detection_loss‚2.492027085274458 .loss/FRt2/detection_loss‚3.746381598360398 1loss/SRqqt1/detection_loss‚2.6831383109092712 epoch‚0 /loss/FBt1/detection_loss‚0.7007454518360948Oô¥$“²‚(:ˆ œ™éÔÜÔm gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚124.661' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.16525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)¹¶á`³‚(:Õ œ™éԣˤo disk.md0.out‚ 117.296875 disk.rbd2.out‚ 0.18017578125 disk.sdb2.in‚ 0.23046875 disk.md0p1.in‚ 1.62890625" disk.nvme0n1.out‚ 3987.33984375 disk.nvme1n1.out‚ 7531.4375 disk.sdb1.in‚0& proc.memory.availableMB‚ 221.796875 network.recv‚436952 disk.nvme1n1.in‚ 5.83203125 cpu‚25.00477248122137 disk.sdb.in‚ 0.23046875! disk.sda2.out‚121.91748046875! disk.sdb2.out‚121.91748046875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.md0p1.out‚ 117.296875 disk.sda2.in‚ 1.3984375 proc.cpu.threads‚17 network.sent‚ 1.416598e+06% disk./.usageGB‚2993.6321868896484 disk.nvme0n1.in‚ 56.55859375 disk.rbd0.out‚ 0.07421875 disk.dm-0.in‚ 62.390625 disk.dm-0.out‚11518.77734375 disk.md0.in‚ 1.62890625 disk.sda.in‚ 1.3984375 disk.sda.out‚121.91748046875 disk.sda1.in‚0$ memory_percent‚98.64625930786133" proc.memory.rssMB‚ 2905.8828125* proc.memory.percent‚17.736101150512695 disk.rbd2.in‚0 disk.sdb1.out‚0) disk./.usagePercent‚83.70320887784246 disk.sdb.out‚121.91748046875z¿áô´‚(:… «™éÔºè÷l gpu.0.gpu‚54.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚178.02' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.505 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷C#!fµ‚(:Û «™éÔÔÁŸm disk.sda.in‚ 1.3984375 disk.md0p1.in‚ 1.62890625 disk.sdb.out‚123.43017578125 disk.md0.in‚ 1.62890625 disk.rbd2.in‚0! disk.sdb2.out‚123.43017578125 network.sent‚ 1.442595e+06 network.recv‚441858 disk.md0p1.out‚ 118.734375" disk.nvme0n1.out‚ 3990.56640625 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd0.out‚ 0.1533203125 disk.sdb1.in‚0" disk.nvme1n1.out‚ 7741.60546875" proc.memory.rssMB‚ 2905.8828125% disk./.usageGB‚2993.6318855285645 disk.sdb1.out‚0 disk.sdb.in‚ 0.23046875 disk.sda2.in‚ 1.3984375 disk.md0.out‚ 118.734375 disk.sdb2.in‚ 0.23046875$ memory_percent‚98.64678382873535 disk.dm-0.out‚ 11732.171875! disk.sda2.out‚123.43017578125 disk.nvme0n1.in‚ 56.55859375 disk.rbd2.out‚ 0.18017578125' proc.memory.availableMB‚ 221.7109375 disk.sda.out‚123.43017578125 disk.dm-0.in‚ 62.390625 disk.nvme1n1.in‚ 5.83203125) disk./.usagePercent‚83.70320045166042 disk.sda1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚17.736101150512695 cpu‚25.004801923974018jòàHj¶‚(:ß º™éÔ‘âÖl! disk.sdb2.out‚125.25732421875 disk.sda1.in‚0 disk.rbd0.out‚ 0.1533203125 disk.sdb.in‚ 0.23046875 disk.sda2.in‚ 1.3984375 disk.nvme1n1.in‚ 5.83203125 disk.rbd2.in‚0 cpu‚25.00540933011552 disk.sda.out‚125.25732421875 disk.sdb.out‚125.25732421875$ memory_percent‚98.64451885223389 proc.cpu.threads‚17 network.recv‚446592 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.rbd2.out‚ 0.22802734375) disk./.usagePercent‚83.72067400667473! disk.nvme0n1.out‚ 4071.1953125" proc.memory.rssMB‚ 2905.8828125 disk.md0p1.in‚ 1.62890625 disk.dm-0.out‚11917.73828125 disk.sdb2.in‚ 0.23046875 disk.sda1.out‚0 disk.nvme0n1.in‚ 56.55859375$ disk./.usageGB‚2994.256824493408 disk.sda.in‚ 1.3984375 disk.sdb1.in‚0 disk.md0p1.out‚ 120.48046875! disk.sda2.out‚125.25732421875" disk.nvme1n1.out‚ 7846.54296875 disk.md0.in‚ 1.62890625 network.sent‚ 1.448865e+06 disk.dm-0.in‚ 62.390625 disk.md0.out‚ 120.48046875( proc.memory.availableMB‚ 222.08203125* proc.memory.percent‚17.736101150512695¡‡û$‘·‚(:† º™éԭˈm gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚179.856' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fÔp»2¸- _wandbruntime‚1152  _runtime‚1152*#‚—Ź¿ & lr-AdamW/pg1‚0.0009130844238301767 & lr-AdamW/pg2‚0.0009130844238301767 trainer/global_step‚1049 $ _timestamp‚1.788497094632922e+09 _runtime‚1152.410152307 _step‚80º©MŸÉºÃ _runtime‚1152.410152307 _step‚80 & lr-AdamW/pg1‚0.0009130844238301767 & lr-AdamW/pg2‚0.0009130844238301767 trainer/global_step‚1049 $ _timestamp‚1.788497094632922e+09P£éñ2»- _wandbruntime‚1152  _runtime‚1152%9YÙ'¼ÓO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1177469400576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚81 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6163545 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚94757980108288 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 # _timestamp‚1.78849709463393e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82990325926384 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚85723896917504 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚84164289142804 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚85723896917504 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚642 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18839487 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1173963216420 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚1152.411564638 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚86902275522560 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚86901348315136 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚86901348315136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12675300 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚95992874775552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚94758004048384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1173055248964 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1178378605056 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26133163 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206480384 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47311955 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26158009 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18838843 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230420480 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12675298 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6164187 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚95993105196032 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚86902275522560 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1178378605056 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21178794 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 trainer/global_step‚1049 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26158009 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12675298 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6164187 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚95993105196032 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚86902275522560 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1178378605056 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21178794 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 trainer/global_step‚1049 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26158009 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚81 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6163545 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚94757980108288 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 # _timestamp‚1.78849709463393e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82990325926384 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚85723896917504 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚84164289142804 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚85723896917504 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚642 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18839487 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1173963216420 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚1152.411564638 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚86902275522560 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚86901348315136 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚86901348315136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12675300 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚95992874775552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚94758004048384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1173055248964 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1178378605056 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26133163 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206480384 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570Q^1"<2¾- _wandbruntime‚1153  _runtime‚1153?dKŒ¿ßM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 _runtime‚1153.967277442 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84244424010834 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚83069343520928 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 _step‚82 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47381820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12687363 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚664 % _timestamp‚1.7884970961878765e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚85805498000384 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1236294565376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚86984088861184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1179500078592 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6169455 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚86985016081920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1178590860800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47356946 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21198952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26182866 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26157994 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 227438592 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6170119 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18857484 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚666 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47381820 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚94848205812736 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84243499003810 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96084524318208 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚94848229752832 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1174172522242 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1179500078592 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18856818 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 251378688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚83069326481568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚86985016081920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚85805516003328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1174172522242 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1179500078592 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18856818 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 251378688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚83069326481568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚86985016081920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚85805516003328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12687363 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚664 % _timestamp‚1.7884970961878765e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚85805498000384 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1236294565376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚86984088861184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1179500078592 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6169455 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚86985016081920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1178590860800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47356946 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21198952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26182866 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26157994 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 227438592 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6170119 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18857484 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚666 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47381820 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚94848205812736 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84243499003810 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96084524318208 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚94848229752832 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22186794 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚89804851208704 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19739791 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6461901 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚88171046125112 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚86941205653584 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884971535709093e+09 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚86941188614224 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27378065 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1233556825600 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚91039317238784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚89804833205760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1234466030080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 trainer/global_step‚1099 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚91038390031360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1294115831808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22186792 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚89804851208704 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚99269265330688 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49564857 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22186792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚99269289270784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚91039317238784 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚89804833205760 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27402911 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1294330700800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27378065 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22186794 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214868992 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚100563619971584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49589705 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238809088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1234466030080 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚91038390031360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1233556825600‹×[ß'ÑÙO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19739791 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6461901 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚88171046125112 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚86941205653584 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884971535709093e+09 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚86941188614224 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27378065 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1233556825600 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚91039317238784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚89804833205760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1234466030080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 trainer/global_step‚1099 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚91038390031360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1294115831808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22186792 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚89804851208704 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚99269265330688 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49564857 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22186792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚99269289270784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚91039317238784 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚89804833205760 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27402911 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1294330700800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27378065 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22186794 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214868992 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚100563619971584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49589705 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238809088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1234466030080 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚91038390031360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1233556825600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22186794 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚89804851208704 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0U5öŸw2Ò- _wandbruntime‚1212  _runtime‚1212è+êç&ÓáM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚648 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88250255986118 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚87020223248128 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1295518601216 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 219049984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚1212.376721193 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22206952 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22206954 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242990080 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22206952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚88251180993142 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19757790 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49609848 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚89886452291584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1230957745014 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚1099 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚89886470294528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1234678285824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚100654790586368 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚87020206208768 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27427768 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1234678285824 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1235587503616 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚89886470294528 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚91121130577408 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1230049777350 _step‚86 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1235587503616 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6467825 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6467177 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49634722 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚91122057798144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚650 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49609848 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27402896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13290615 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27402896’1Ãë&ÔåM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚91122057798144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19757790 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49609848 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚89886452291584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1230957745014 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚1099 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚89886470294528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1234678285824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚100654790586368 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚87020206208768 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27427768 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1234678285824 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1235587503616 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚89886470294528 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚91121130577408 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1230049777350 _step‚86 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1235587503616 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6467825 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6467177 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49634722 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚91122057798144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚650 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49609848 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27402896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13290615 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27402896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚648 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88250255986118 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚87020223248128 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1295518601216 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 219049984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚1212.376721193 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22206952 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22206954 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242990080 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22206952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚88251180993142 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚92177803107420 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚105133768878080 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1353218324992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚90892085380784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23194792 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51817759 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚93885787496960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23194794 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1289644250624 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚660 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚95176358955008 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚95176358955008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1290553455104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13881800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 _runtime‚1264.179124666 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚93885805499904 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1289644250624 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1285717726636 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚103780550553088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚95175431747584 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚93885787496960 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1353431096832 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚95175431747584 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973>}ç‰ã'åÝO A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚95176358955008 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚95176358955008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1290553455104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13881800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 _runtime‚1264.179124666 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚93885805499904 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1289644250624 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1285717726636 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚103780550553088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚95175431747584 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚93885787496960 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1353431096832 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚95175431747584 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20640559 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236711936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚92176878100604 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚90892068341424 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚92177803107420 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚105133768878080 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1353218324992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚90892085380784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23194792 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51817759 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚93885787496960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23194794 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1289644250624 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚660 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658YÓ[ 2æ- _wandbruntime‚1265  _runtime‚1265«Ñ9CçãM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28672670 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28672670 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _step‚90 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13893865 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚90971085935968 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1290765710848 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚90971102975328 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚93967406582784 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23214952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20658543 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚92257937975450 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚93967424585728 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚105225425442304 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 F@DeviceStatsMonitor.on_train_bÿد‹¦atch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51862750 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13893863 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚95259099514368 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28647798 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _runtime‚1265.114391751 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1285927032458 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23214954 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚93967406582784 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚95258172293632 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚92257012968426 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚655 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236698624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚1149 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚93967424585728 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51887624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51887624 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6764025 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884972073365343e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20657888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚95259099514368 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23214954 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28647798 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚103870800197632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1291674928640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚105225188743680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1354625244672 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6764678 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23214952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212758528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1286835000122 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚653 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0ü¼j.í&èçM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1290765710848 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚90971102975328 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚93967406582784 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23214952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20658543 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚92257937975450 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚93967424585728 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚105225425442304 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51862750 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13893863 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚95259099514368 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28647798 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _runtime‚1265.114391751 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1285927032458 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23214954 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚93967406582784 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚95258172293632 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚92257012968426 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚655 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236698624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚1149 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚93967424585728 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51887624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51887624 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6764025 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884972073365343e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20657888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚95259099514368 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23214954 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28647798 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚103870800197632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1291674928640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚105225188743680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1354625244672 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6764678 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23214952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212758528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1286835000122 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚653 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28672670 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28672670 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _step‚90 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13893865 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚90971085935968Z6­Ö·2é- _wandbruntime‚1265  _runtime‚1265e¿ZþVêÐ /loss/FRt1/detection_loss‚2.4206225467440223 2loss/SRqqt2/assignment_loss‚2.7239670972613728 .loss/FRt3/detection_loss‚1.687106535048552 0loss/SRqqt1/detection_loss‚2.311063514514403 0loss/FRt1/assignment_loss‚2.9842380246808453 /loss/FRt4/assignment_loss‚4.556345758766963 2loss/SRqqt1/assignment_loss‚1.7917464355176145 1loss/SRqqt4/assignment_loss‚2.189637068245146 /loss/FBt1/assignment_loss‚0.206645737064861 1loss/FBt2/assignment_loss‚0.37553400745465687 trainer/global_step‚1149 % _timestamp‚1.7884972073393455e+09 .loss/FBt2/detection_loss‚4.152081378868648 /loss/FBt4/detection_loss‚3.1357814898857703 _runtime‚1265.116337291 /loss/FRt3/assignment_loss‚2.055419622172772 2loss/SRqqt3/assignment_loss‚1.4935603961006765 .loss/FRt4/detection_loss‚2.885775015271943 1loss/SRqqt4/detection_loss‚2.4526993334293365 %loss/total_loss‚53.32218120647801 epoch‚0 1loss/SRqqt3/detection_loss‚1.0369907997958703 /loss/FBt3/detection_loss‚3.5339575012524924 /loss/FRt2/detection_loss‚3.0362887520056505 0loss/SRqqt2/detection_loss‚2.497694518636255 .loss/FRt2/assignment_loss‚4.67116669508127 1loss/FBt3/assignment_loss‚0.10078642005100846 1loss/FBt4/assignment_loss‚0.24289262253576174 /loss/FBt1/detection_loss‚0.7801799360921176 _step‚91lb¯ÑZëÔ /loss/FRt3/assignment_loss‚2.055419622172772 2loss/SRqqt3/assignment_loss‚1.4935603961006765 .loss/FRt4/detection_loss‚2.885775015271943 1loss/SRqqt4/detection_loss‚2.4526993334293365 %loss/total_loss‚53.32218120647801 epoch‚0 1loss/SRqqt3/detection_loss‚1.0369907997958703 /loss/FBt3/detection_loss‚3.5339575012524924 /loss/FRt2/detection_loss‚3.0362887520056505 0loss/SRqqt2/detection_loss‚2.497694518636255 .loss/FRt2/assignment_loss‚4.67116669508127 1loss/FBt3/assignment_loss‚0.10078642005100846 1loss/FBt4/assignment_loss‚0.24289262253576174 /loss/FBt1/detection_loss‚0.7801799360921176 _step‚91 /loss/FRt1/detection_loss‚2.4206225467440223 2loss/SRqqt2/assignment_loss‚2.7239670972613728 .loss/FRt3/detection_loss‚1.687106535048552 0loss/SRqqt1/detection_loss‚2.311063514514403 0loss/FRt1/assignment_loss‚2.9842380246808453 /loss/FRt4/assignment_loss‚4.556345758766963 2loss/SRqqt1/assignment_loss‚1.7917464355176145 1loss/SRqqt4/assignment_loss‚2.189637068245146 /loss/FBt1/assignment_loss‚0.206645737064861 1loss/FBt2/assignment_loss‚0.37553400745465687 trainer/global_step‚1149 % _timestamp‚1.7884972073393455e+09 .loss/FBt2/detection_loss‚4.152081378868648 /loss/FBt4/detection_loss‚3.1357814898857703 _runtime‚1265.116337291[†òÊu“ì‚(:ˆ ÁšéÔòÖˆl gpu.0.gpu‚53.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚171.501' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.87525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}ÐQí‚(:Æ ÁšéÔº´ÿm disk.sdb.out‚ 140.115234375 disk.sda2.out‚ 140.115234375 disk.nvme0n1.in‚ 56.55859375 disk.rbd2.out‚ 0.35400390625 disk.sdb1.out‚0 network.recv‚485524 disk.nvme0n1.out‚4723.25 disk.sda1.in‚0* proc.memory.percent‚17.736101150512695 network.sent‚ 1.600287e+06 disk.dm-0.out‚13352.44140625 disk.sda2.in‚ 1.3984375 disk.sdb2.in‚ 0.23046875 disk.rbd0.in‚0' proc.memory.availableMB‚ 221.4609375 disk.md0.in‚ 1.62890625 disk.sda1.out‚0) disk./.usagePercent‚83.68889940751355% disk./.usageGB‚2993.1204109191895 disk.md0.out‚134.75 disk.sda.out‚ 140.115234375# memory_percent‚98.6483097076416 disk.dm-0.in‚ 65.56640625 disk.rbd2.in‚0 disk.sdb1.in‚0 proc.cpu.threads‚17 disk.sda.in‚ 1.3984375 disk.rbd0.out‚ 0.279296875 disk.md0p1.out‚134.75 disk.sdb.in‚ 0.23046875 disk.nvme1n1.in‚ 9.0078125" disk.nvme1n1.out‚ 8629.19140625 disk.sdb2.out‚ 140.115234375 disk.md0p1.in‚ 1.62890625" proc.memory.rssMB‚ 2905.8828125 cpu‚25.007293991852695·ÑÞî‚(:… КéÔÁÒík gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.22' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.055 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò³ñbï‚(:× КéÔÙáðl) disk./.usagePercent‚83.69609792682644 disk.md0p1.out‚ 136.44140625! disk.nvme1n1.out‚ 8773.2109375' proc.memory.availableMB‚ 221.4453125% disk./.usageGB‚2993.3778648376465 disk.dm-0.in‚ 65.56640625 disk.sdb.in‚ 0.23046875 disk.sda2.in‚ 1.3984375 disk.nvme1n1.in‚ 9.0078125 disk.md0.out‚ 136.44140625 disk.rbd2.out‚ 0.35400390625" proc.memory.rssMB‚ 2905.8828125 disk.sdb1.out‚0 disk.md0p1.in‚ 1.62890625 disk.rbd2.in‚0 disk.sda.in‚ 1.3984375 disk.sda.out‚ 141.857421875 disk.sda1.out‚0 network.recv‚491165 disk.sdb1.in‚0 disk.sda2.out‚ 141.857421875 disk.sdb2.in‚ 0.23046875 disk.rbd0.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sdb.out‚ 141.857421875 network.sent‚ 1.636678e+06 disk.dm-0.out‚13594.10546875 disk.nvme0n1.in‚ 56.55859375" disk.nvme0n1.out‚ 4820.89453125$ memory_percent‚98.64840507507324* proc.memory.percent‚17.736101150512695 cpu‚25.00725536925539 disk.rbd0.out‚ 0.279296875 disk.md0.in‚ 1.62890625 disk.sdb2.out‚ 141.857421875Ñññ’ð‚(:‡ ßšéÔ¼‚âl gpu.0.gpu‚5.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚176.461' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.11525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 J‡ìhñ‚(:Ý ßšéÔ¢‘«m network.sent‚ 1.642606e+06 network.recv‚495385 disk.rbd2.in‚0 disk.dm-0.out‚13694.97265625 disk.sdb.out‚143.36865234375" disk.nvme0n1.out‚ 4917.17578125 disk.md0.out‚ 137.8984375 disk.sdb2.in‚ 0.23046875 disk.sda1.out‚0( proc.memory.availableMB‚ 221.44921875$ disk./.usageGB‚2993.633159637451 disk.sdb1.out‚0$ memory_percent‚98.64838123321533" proc.memory.rssMB‚ 2905.8828125 disk.md0p1.in‚ 1.62890625 disk.md0p1.out‚ 137.8984375 disk.sda.out‚143.36865234375 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.dm-0.in‚ 65.56640625 disk.sdb.in‚ 0.23046875 disk.nvme0n1.in‚ 56.55859375 disk.md0.in‚ 1.62890625 disk.rbd2.out‚ 0.35400390625! disk.sdb2.out‚143.36865234375 disk.sda1.in‚0! disk.sda2.out‚143.36865234375 disk.nvme1n1.in‚ 9.0078125 disk.sda.in‚ 1.3984375 disk.rbd0.out‚ 0.279296875 cpu‚25.007719306208557 proc.cpu.threads‚17) disk./.usagePercent‚83.70323607627816 disk.sda2.in‚ 1.3984375 disk.nvme1n1.out‚ 8777.796875* proc.memory.percent‚17.736101150512695éb›Ð2ò- _wandbruntime‚1317  _runtime‚1317×Í}çÅó¿ & lr-AdamW/pg2‚0.0010436494034055117 trainer/global_step‚1199 $ _timestamp‚1.788497259409315e+09 _runtime‚1317.186676843 _step‚92 & lr-AdamW/pg1‚0.0010436494034055117ÔF‡JÉôà _runtime‚1317.186676843 _step‚92 & lr-AdamW/pg1‚0.0010436494034055117 & lr-AdamW/pg2‚0.0010436494034055117 trainer/global_step‚1199 $ _timestamp‚1.788497259409315e+09\£ñÊ2õ- _wandbruntime‚1317  _runtime‚1317ƒýxY"öØO YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1345731675648 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29867869 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108291835775488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1340687014288 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99312473463808 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚97966741788160 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚96184560089728 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚94842948068624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24202792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚94842965107984 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24202794 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7054645 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29892715 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚99312473463808 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚97966759791104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14485048 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96183635082912 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1346640880128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚97966759791104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21539693 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245100544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29892715 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108291859715584 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1412544866304 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚1199 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚99313400671232 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1346640880128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221160448 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 1317.18813352 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7055321 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14485050 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚97966741788160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚678 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424Zå/qâ'÷ÜO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1345731675648 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29867869 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108291835775488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1340687014288 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99312473463808 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚97966741788160 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚96184560089728 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚94842948068624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24202792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚94842965107984 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24202794 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7054645 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29892715 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚99312473463808 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚97966759791104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14485048 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96183635082912 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1346640880128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚97966759791104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21539693 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245100544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29892715 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108291859715584 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1412544866304 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚1199 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚99313400671232 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1346640880128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221160448 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 1317.18813352 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7055321 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14485050 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚97966741788160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚678 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640] :X2ø- _wandbruntime‚1318  _runtime‚1318ZŒ‚Ýé&ùãM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54115652 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24222952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1346853135872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204369920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24222954 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1413733182464 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 trainer/global_step‚1199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21558365 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24222954 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚98048378876928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54115652 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚661 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54140526 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚109795590292480 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚108382061479936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29917572 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96263769950734 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚99396141230592 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29892700 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21557702 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚99395214009856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 % _timestamp‚1.7884972605455897e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1347762353664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1413528812544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚98048360873984 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1347762353664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54140526 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 _runtime‚1318.323648015 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570ëk‰í&úçM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24222952 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7061250 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7060589 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1341804287566 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚98048378876928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14497115 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 _runtime‚1318.323648015 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54115652 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24222952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1346853135872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204369920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24222954 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1413733182464 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 trainer/global_step‚1199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21558365 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24222954 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚98048378876928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54115652 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚661 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54140526 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚109795590292480 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚108382061479936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29917572 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96263769950734 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚99396141230592 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29892700 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21557702 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚99395214009856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 % _timestamp‚1.7884972605455897e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1347762353664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1413528812544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚98048360873984 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1347762353664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54140526 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 F¯>k¬bf CDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak"trainer/global_step2:JPkˆäy­bt QDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JPŠ`m q®bl IDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JP#J}d¯b_ DeviceStatsMonitor.on_validation_batch_start/segment.all.freed"trainer/global_step2:JP*«eXvìbq NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JP¨4–öríbm JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JPF˜Þyîbt QDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JPŸ"sžzïbu RDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPya‚wðbr ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JPo+39uñbp MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JP(ïòqsòbn KDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated"trainer/global_step2:JP¥>jgpóbk HDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated"trainer/global_step2:JPÞÉfôba >DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries"trainer/global_step2:JP+.ƒ¢tõbo LDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JPOÉ#àqöbl IDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated"trainer/global_step2:JP Sݲm÷bh EDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed"trainer/global_step2:JP(Äõºpøbk HDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed"trainer/global_step2:JP»™–rùbm JDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JPíîájúbe BDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak"trainer/global_step2:JPÍ·Ê@kûbf CDeviceStatsMonitor.on_validation_batch_start/allocation.all.current"trainer/global_step2:JPº^Úítübo LDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated"trainer/global_step2:JPLîoýbj GDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current"trainer/global_step2:JP”(,ªmþbh EDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed"trainer/global_step2:JP æclÿbg DDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak"trainer/global_step2:JP]Ã…µs€bn KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JP™™…jbe BDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated"trainer/global_step2:JPÒ¨ž w‚br ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JPý=vË2ƒ- _wandbruntime‚1390  _runtime‚1390Ì5–!„ÈT NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚55715681 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 180164096 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚15074238 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1231 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 156224000 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1185 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1185 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1185 trainer/global_step‚1208 _runtime‚1390.678044172 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚98104020207440 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14401142784 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 1381873147904 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚102823608240640 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚101441735092736 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚111951202955776 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 681064172 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 1381190511104 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚30572890 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚615 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚25162716 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 409073152 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 14134971392 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚19927 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚25162718 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚216 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚22458185 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1268 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allÆÌ:¸ocated‚1185 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 682636800 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚102822907600896 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3725 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189001216 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13684408973 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚19927 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚30552965 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 698103532 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3725 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 1448474266112 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 1376024998304 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚30751 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚22458401 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚15074236 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14401142784\¥ØóR*…ÌT cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 1376706062476 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚101441735092736 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 682636800 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 14134971392 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚101441717089792 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 1448630490112 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 12553587712 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚570 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚102822907600896 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 13002869760 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚55715681 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚30552965 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1195376640 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚113399857385984 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚570 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13205766144 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚19925 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 1381190511104 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 1381873147904 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚102823608240640 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚101441735092736 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚111951202955776 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 681064172 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 1381190511104 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚30572890 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚615 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚25162716 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 409073152 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 14134971392 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚19927 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚25162718 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚216 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚22458185 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1268 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1185 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 682636800 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚102822907600896 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3725 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189001216 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13684408973 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚19927 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚30552965 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 698103532 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3725 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 1448474266112 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 1376024998304 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚30751 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚22458401 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚15074236 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14401142784 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚55715681 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 180164096 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚15074238 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1231 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 156224000 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1185 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1185 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1185 trainer/global_step‚1208 _runtime‚1390.678044172 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚98104020207440 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14401142784 FDeviceStatsMonitor.on_validation_batch_end/segment.all.current"trainer/global_step2:JPC¿jv¡bq NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JP¯–JËv¢bq NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JP··2n£bi FDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current"trainer/global_step2:JP-Síu¤bp MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JPn·‚Li¥bd ADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak"trainer/global_step2:JP¼™Q.e¦b` =DeviceStatsMonitor.on_validation_batch_end/active.all.current"trainer/global_step2:JPáš6wr§bm JDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current"trainer/global_step2:JPd 3ïr¨bm JDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated"trainer/global_step2:JP}ÄéSi©bd ADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak"trainer/global_step2:JPËj—iªbd ADeviceStatsMonitor.on_validation_batch_end/allocation.all.current"trainer/global_step2:JPž­ù™r«bm JDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JPĈõ3h¬bc @DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated"trainer/global_step2:JPÈDøcn­bi FDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated"trainer/global_step2:JPýØ w®br ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP:wó’l¯bg DDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak"trainer/global_step2:JPÎìso°bj GDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JPæŒÐûd±b_ 1lÝbg DDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current"trainer/global_step2:JPQ¹yjÞbe BDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed"trainer/global_step2:JPɃ1ätßbo LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JPІ‚càb^ ;DeviceStatsMonitor.on_validation_batch_end/active.all.freed"trainer/global_step2:JPiͬKlábg DDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed"trainer/global_step2:JP2ƒwlâbg DDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current"trainer/global_step2:JP2 ×|jãbe BDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed"trainer/global_step2:JPÂìwäbr ODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JP£#õ,oåbj GDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated"trainer/global_step2:JP&Ë< gæbb ?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated"trainer/global_step2:JP~„76içbd ADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak"trainer/global_step2:JPG:\·pèbk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JP^øÃ/pébk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JPùžpLnêbi FDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed"trainer/global_step2:JP }æZvëbq NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPý¾º{eìb` =DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections"trainer/global_step2:JP1¿:ríbm JDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JPßáj8bîb] :DeviceStatsMonitor.on_validation_batch_end/active.all.peak"trainer/global_step2:JP¢8åmïbh EDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated"trainer/global_step2:JPË`ë”kðbf CDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed"trainer/global_step2:JPZ•*äsñbn KDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JP“\q{xòbs PDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP²¦móbh EDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current"trainer/global_step2:JP ·ætôbo LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JP‰ðÒÌrõbm JDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JPü¾,Øtöbo LDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current"trainer/global_step2:JPyTðçs÷bn KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JP’7Ù…tøbo LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JP~¹mùbh EDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current"trainer/global_step2:JP©E½júbe BDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak"trainer/global_step2:JPÀÒÓrûbm JDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPŒ¹½iübd ADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed"trainer/global_step2:JPfW‹Rsýbn KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JP|”°fþba >DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak"trainer/global_step2:JP¨’ˆ[ÿbV 3DeviceStatsMonitor.on_validation_batch_end/num_ooms"trainer/global_step2:JP2T¸6c€b^ ;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak"trainer/global_step2:JPÿˆ<š2- _wandbruntime‚1391  _runtime‚1391F"ó-‚ÒR P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 14134971392 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚55739971 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 1377194273360 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14134971392 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 1376513209188 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 _step‚97 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 1382362998784 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698103532 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚30508 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚19927 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚25178194 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚19927 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1185 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚25178194 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14401142784 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚30751 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1185 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚30581704 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚216 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 WDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚99530942482452 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚15083800 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 700639744 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 % _timestamp‚1.7884973341172383e+09 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 682636800 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚113456434626560 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚99531640585984 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚615 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚5705­8\)ƒÖR TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698103532 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚30508 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚19927 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚25178194 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚19927 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1185 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚25178194 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14401142784 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚30751 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1185 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚30581704 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚216 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 WDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚99530942482452 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚15083800 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 700639744 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 % _timestamp‚1.7884973341172383e+09 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 682636800 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚113456434626560 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚99531640585984 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚615 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚570 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 14134971392 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚55739971 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 1377194273360 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14134971392 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 1376513209188 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 _step‚97 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 1382362998784 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640aýÂæ#k„‚(:à ¹›éÔ»¶©p disk.sda2.in‚ 1.3984375 disk.rbd2.in‚0* proc.memory.percent‚18.228840827941895% disk./.usageGB‚2994.1315116882324 disk.sda1.out‚0 disk.sdb1.out‚0 network.recv‚520670! disk.sda2.out‚153.17919921875 disk.nvme1n1.in‚ 9.0078125! disk.nvme1n1.out‚ 9250.7421875 disk.sda.in‚ 1.3984375 disk.sdb1.in‚0$ memory_percent‚99.14166927337646 disk.rbd0.in‚0 disk.dm-0.out‚14670.02734375 disk.sdb.in‚ 0.23046875 disk.nvme0n1.in‚ 56.55859375 disk.sdb2.in‚ 0.23046875 disk.md0p1.out‚ 147.32421875 disk.dm-0.in‚ 65.56640625 disk.sdb.out‚153.17919921875 disk.rbd2.out‚ 0.35400390625! disk.sdb2.out‚153.17919921875 proc.cpu.threads‚17( proc.memory.availableMB‚ 140.62890625# proc.memory.rssMB‚ 2986.61328125 disk.md0.in‚ 1.62890625 disk.sda.out‚153.17919921875 disk.rbd0.out‚ 0.279296875 disk.sda1.in‚0 network.sent‚ 1.72006e+06) disk./.usagePercent‚83.71717020819453" disk.nvme0n1.out‚ 5419.28515625 disk.md0.out‚ 147.32421875 cpu‚25.19158494226656 disk.md0p1.in‚ 1.62890625é¥Ö“…‚(:ˆ ¹›éÔ–òÀo gpu.0.gpu‚52.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚117.193' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.29825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0È sN†bI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPñÆœóH‡bC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPœ¶²NˆbI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP¼OpðN‰bI &Purity/0FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP"¦¯ôNŠbI &Purity/1FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP7‡ñâN‹bI &Purity/3FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPQào,;Œb6 jet/accuracy_6_of_6"trainer/global_step2:JPÙÙïHbC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP¿ÜNŽbI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPÎÒ©NbI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPžb•NbI &Purity/0FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP'CÐJ‘bE "Purity/1FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP¢ îJH’bC Purity/4FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP·XðH“bC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP¹ý·EN”bI &Purity/*FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPfÀŸnN•bI &Purity/1FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPšg}íH–bC Purity/2FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPÐr$H—bC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP~t'H˜bC Purity/2FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPlƒ¢H™bC Purity/3FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPþ0ÌóNšbI &Purity/3FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPeƒH›bC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP–dNœbI &Purity/3FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPÁÚ—§NbI &Purity/3FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPœÇOÄJžbE "Purity/3FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP=WáÎNŸbI &Purity/3FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP÷ôEÄ; b6 jet/accuracy_1_of_1"trainer/global_step2:JP¯ufH¡bC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP¿Ô`]N¢bI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP@í®N£bI &Purity/0FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPvÅn­J¤bE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP‡eÃÙH¥bC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPÃKZH¦bC Purity/2FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPÙ?Ç›N§bI &Purity/3FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP¨°í“N¨bI &Purity/4FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP! H©bC Purity/*FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPA?IDHªbC Purity/0FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPëýJ«bE "Purity/1FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP‰ºJ¬bE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP•zN­bI &Purity/3FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPA0ÙMH®bC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPLJքH¯bC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP­qÀÐN°bI &Purity/0FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP?4ípNÏbI &Purity/1FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP‘uöJÐbE "Purity/*FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPiÏ¡JÑbE "Purity/2FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPû¡£°HÒbC Purity/2FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPCLðNÓbI &Purity/3FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP‚{ð@Ôb; particle/accuracy_4_of_7"trainer/global_step2:JPH)HÕbC Purity/0FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPÑ :"JÖbE "Purity/2FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP#~3HJ×bE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP¹º>YJØbE "Purity/1FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPruJÙbE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPçå$HÚbC Purity/0FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPÆãÛƒJÛbE "Purity/0FRt1SRqqt3FBt/event_purity"trainer/global_step2:JP~YÎNÜbI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPÇôܨNÝbI &Purity/1FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP¯a5€JÞbE "Purity/1FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPUv™ÆNßbI &Purity/1FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP[E/ôJàbE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPX\€HábC Purity/2FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPOQ&HâbC Purity/2FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPFoÌ¿NãbI &Purity/2FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPNèÔHäbC Purity/*FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPãÅòºNåbI &Purity/0FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP >ÎJæbE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP©á!ÒNçbI &Purity/0FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPû'‘ŽNèbI &Purity/3FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP·k÷NébI &Purity/4FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPhÝ„NêbI &Purity/4FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPiL{ZJëbE "Purity/*FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP“:¡HìbC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPzöÜ~NíbI &Purity/0FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP—@¨ÅJîbE "Purity/2FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP{/¬¾NïbI &Purity/4FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPÑs6@ðb; particle/accuracy_4_of_5"trainer/global_step2:JPYcÆWJñbE "Purity/*FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPŸJ5ªHòbC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPÙ#)îJóbE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP̾ 3NôbI &Purity/2FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPG½¸ÊHõbC Purity/*FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JP»TîJöbE "Purity/2FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÊÍÆN÷bI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP ê ,HøbC Purity/1FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPŒ]¯åHùbC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP¤ÓSJúbE "Purity/1FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPë®nHûbC Purity/4FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP’ú'7JübE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPËœé³HýbC Purity/2FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP<š¤Ö;þb6 jet/accuracy_4_of_5"trainer/global_step2:JPÒ}y~JÿbE "Purity/0FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPo4|…@€ b; particle/accuracy_2_of_7"trainer/global_step2:JPë"û H bC Purity/3FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP…FuH‚ bC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPO­[hNƒ bI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP—ÑpN„ bI &Purity/1FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPǧö„N… bI &Purity/2FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP œ¨Å;† b6 jet/accuracy_1_of_5"trainer/global_step2:JPïœJ‡ bE "Purity/1FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÞ¤WæHˆ bC Purity/3FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPXX/H‰ bC Purity/3FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP iV5HŠ bC Purity/1FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP;~ÆN‹ bI &Purity/4FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPk [®JŒ bE "Purity/3FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPô@mÏJ bE "Purity/0FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP8YËxNŽ bI &Purity/0FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPmaRJ bE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPʼn”ÈJ bE "Purity/*FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPASeÈN‘ bI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPÐáôH’ bC Purity/1FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPÉ,UJ“ bE "Purity/1FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPô‡— N” bI &Purity/1FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP¦*O”J• bE "Purity/3FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP­Ñ@– b; particle/accuracy_7_of_7"trainer/global_step2:JP8ΰH— bC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP*«½¯H˜ bC Purity/1FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP‘ šzN™ bI &Purity/1FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP>¾ÍHš bC Purity/1FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JP€;.N› bI &Purity/1FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP¥wHà bC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP%:˜¬Já bE "Purity/0FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP(ÙhJâ bE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP«wTJã bE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP\ŽóNä bI &Purity/1FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPÄ’îVNå bI &Purity/3FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPe"µJæ bE "Purity/0FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP +EzJç bE "Purity/*FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÚÛÓùJè bE "Purity/*FRt2SRqqt3FBt/event_purity"trainer/global_step2:JPZ(Né bI &Purity/3FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPƒfáNê bI &Purity/3FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP­+§„Hë bC Purity/2FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JP%@ÃèJì bE "Purity/4FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPC”ý"Jí bE "Purity/2FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPÝa…ëHî bC Purity/2FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPÒ"©aNï bI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPü&³Jð bE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP|‘™ Jñ bE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP/S=ÎNò bI &Purity/4FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPK—ß«Jó bE "Purity/3FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPú¨2Jô bE "Purity/1FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JP‚>îfNõ bI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPã/E3Hö bC Purity/*FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPe˜JúH÷ bC Purity/*FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP²Í üHø bC Purity/2FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPNkPUJù bE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPóÐ(ÄHú bC Purity/2FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPug' Hû bC Purity/2FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPà"¹ÇJü bE "Purity/3FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPMNæHý bC Purity/*FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPZB@Hþ bC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP¤—.Nÿ bI &Purity/4FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPû·C÷N€ bI &Purity/4FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP<ºì©J bE "Purity/3FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP¤P;4H‚ bC Purity/4FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPÑK}çHƒ bC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP¿_ON„ bI &Purity/0FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPbTšœN… bI &Purity/3FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPO(O¨J† bE "Purity/3FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPRÒ -N‡ bI &Purity/*FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPÏ*_¼Jˆ bE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP…ÌöÈN‰ bI &Purity/0FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP4sNŠ bI &Purity/3FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPçõvN‹ bI &Purity/3FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPž_q¼JŒ bE "Purity/0FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP]ä»J bE "Purity/*FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPŠ,JŸJŽ bE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP dV6H bC Purity/*FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JP”5Ê€J bE "Purity/3FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPöÿ@$J‘ bE "Purity/4FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP:7¬¥J’ bE "Purity/*FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP6N[•N“ bI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP†Ÿë}H” bC Purity/*FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP(ä´H• bC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPi…zN– bI &Purity/1FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPO.4²N— bI &Purity/3FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPŠTÿN˜ bI &Purity/4FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP8—x;™ b6 jet/accuracy_2_of_4"trainer/global_step2:JPâ(!Jš bE "Purity/*FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPêªÞH› bC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP U5ZNœ bI &Purity/1FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPÞô%J bE "Purity/3FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP¶£FqHž bC Purity/3FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP0I¸HŸ bC Purity/3FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP‹¶.;  b6 jet/accuracy_1_of_7"trainer/global_step2:JP%BÃ2J¡ bE "Purity/0FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPí$µJ¢ bE "Purity/0FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP?ž(ÿJ£ bE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPçTxÙJ¤ bE "Purity/2FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP€&ì¦H¥ bC Purity/1FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPX/¶›H¦ bC Purity/4FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP`=ìðJ§ bE "Purity/0FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP ‹H¨ bC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP'‰áæ;© b6 jet/accuracy_3_of_6"trainer/global_step2:JPT§pÔJª bE "Purity/1FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPØ|ÀÛH« bC Purity/2FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JP› ˜1J¬ bE "Purity/1FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPO4úÏN­ bI &Purity/2FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPŵ(dN® bI &Purity/2FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPÓ)“w@¯ b; particle/accuracy_1_of_2"trainer/global_step2:JP©äM«@° b; particle/accuracy_5_of_7"trainer/global_step2:JP`Î-N± bI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPCÈ3N² bI &Purity/1FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP‘ì„J³ bE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP§‘ÕbN´ bI &Purity/2FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP·¡Ñ%Nµ bI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPèÒqÐ<¶ b7 particle/specificity"trainer/global_step2:JP›Z¿H· bC Purity/2FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP*Ð …J¸ bE "Purity/3FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÀþ©J¹ bE "Purity/0FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPN¾Í÷Jº bE "Purity/1FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP8oÖ9N» bI &Purity/2FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPƒ©J¼ bE "Purity/3FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPŽts‡J½ bE "Purity/*FRt4SRqqt1FBt/event_purity"trainer/global_step2:JPM'QçN¾ bI &Purity/*FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP1ÂAN¿ bI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP á.ç@À b; particle/accuracy_3_of_7"trainer/global_step2:JPö õzHÁ bC Purity/3FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP{{7’N bI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPL`¡•Jà bE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPXê¶HÄ bC Purity/1FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP9=ÙNÅ bI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP2ù•RNÆ bI &Purity/1FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPëU"~NÇ bI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP»—àJÈ bE "Purity/2FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPVXð]HÉ bC Purity/0FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JP"ÄßÒHÊ bC Purity/*FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPÍâ ¨JË bE "Purity/4FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP½¹´JÌ bE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP•©ˆ/HÍ bC Purity/1FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP¶QF`HÎ bC Purity/2FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JP˜î¾HÏ bC Purity/3FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP(¨ðJÐ bE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP‹2úJÑ bE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP§ÏÆ;JÒ bE "Purity/3FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPcÆ¡HÓ bC Purity/3FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPaAäúHÔ bC Purity/3FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPL!QîHÕ bC Purity/1FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPý «[Nð bI &Purity/3FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP"è¶p;ñ b6 jet/accuracy_2_of_2"trainer/global_step2:JP†Îé×Hò bC Purity/0FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JP£ òJó bE "Purity/1FRt*SRqqt4FBt/event_purity"trainer/global_step2:JPykNô bI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPëÆd¦Nõ bI &Purity/2FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPæ€@3Jö bE "Purity/*FRt0SRqqt4FBt/event_purity"trainer/global_step2:JP³þÏJ÷ bE "Purity/*FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPØDJJø bE "Purity/2FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP(µËNù bI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP’? Hú bC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPÀÛ˜´;û b6 jet/accuracy_3_of_3"trainer/global_step2:JP¨S!ù;ü b6 jet/accuracy_1_of_4"trainer/global_step2:JP7ï¥ÚJý bE "Purity/*FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP&Ó?'Hþ bC Purity/0FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP3ñ[Jÿ bE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPù¿áH€ bC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP)”+9N bI &Purity/2FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPÑÐVH‚ bC Purity/*FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP‹å!ŒJƒ bE "Purity/0FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP¸z{ÓJ„ bE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPè'iH… bC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPÿ*ÏH† bC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPùPî'J‡ bE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP)B•Nˆ bI &Purity/2FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP#1:‰N‰ bI &Purity/3FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPÎX‚^NŠ bI &Purity/4FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPô(bJ‹ bE "Purity/3FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPQžýJŒ bE "Purity/3FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPì°€H bC Purity/1FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JP‚ïJJŽ bE "Purity/*FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JP|"„uJ bE "Purity/*FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JP%*~J bE "Purity/3FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPç¥ð6H‘ bC Purity/4FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPÄŽù¥N’ bI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPÓš8ØN“ bI &Purity/0FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPçE:ºN” bI &Purity/2FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPÙÞ<®H• bC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP‘†rH– bC Purity/2FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP1}ÎH— bC Purity/2FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPT_CˆN˜ bI &Purity/3FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPV_iTH™ bC Purity/3FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP+1¬rJš bE "Purity/1FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPˆz¤J› bE "Purity/2FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP‰oNœ bI &Purity/3FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPAZ™H bC Purity/3FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPmFwwJž bE "Purity/1FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP=#€·HŸ bC Purity/3FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP¦µ†N  bI &Purity/*FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP´,¦N¡ bI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP“œ­=N¢ bI &Purity/4FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP…‘ÄH£ bC Purity/*FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JP°ë™¼H¤ bC Purity/*FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JP.Æ%H¥ bC Purity/3FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPUÞJ¦ bE "Purity/0FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPOÇD]N§ bI &Purity/2FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP#N¨ bI &Purity/*FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP—ïí2H© bC Purity/1FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JP{ÜhJª bE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPNR*›H« bC Purity/3FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPÛ … N¬ bI &Purity/*FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPNܹN­ bI &Purity/*FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPóò„N® bI &Purity/*FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPaè•¥N¯ bI &Purity/0FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP*‚ÕïH° bC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPô20ß;± b6 jet/accuracy_3_of_5"trainer/global_step2:JPÀ.bH² bC Purity/0FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JPÐu,8J³ bE "Purity/4FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPøg¯N´ bI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPÔÉRVHµ bC Purity/1FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP£¬ÒÙH¶ bC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP3Íìù@· b; particle/accuracy_1_of_4"trainer/global_step2:JP]wNiJ¸ bE "Purity/3FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPð‘MJ¹ bE "Purity/3FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP¬V©ýNº bI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPÜáÄN» bI &Purity/*FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP¹ŽsGH¼ bC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP†* N½ bI &Purity/4FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP±/wH¾ bC Purity/3FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP Y„ÇJ¿ bE "Purity/4FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP·åðNÀ bI &Purity/2FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPf»šNÁ bI &Purity/4FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPð¼ÂH bC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPéùB­Nà bI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP1JÄ bE "Purity/1FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPµµžJÅ bE "Purity/3FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP–†zÃJÆ bE "Purity/0FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPõQJÇ bE "Purity/*FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPP2™rJÈ bE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP¹´¶NÉ bI &Purity/2FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPÚ‡,RNÊ bI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP­>/HË bC Purity/*FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JPRs8';Ì b6 jet/accuracy_6_of_7"trainer/global_step2:JP9°wqJÍ bE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP±©ãHÎ bC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP´_NÏ bI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPº‚ÑùHÐ bC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPîx=ÐJÑ bE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPÒ'ºQNÒ bI &Purity/4FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP½ðzGÓ bB validation_average_jet_accuracy"trainer/global_step2:JPs2O9HÔ bC Purity/1FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPà7ÛO@Õ b; particle/accuracy_2_of_3"trainer/global_step2:JPôÄJÖ bE "Purity/*FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPd>BŸH× bC Purity/1FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JP:ú}ÏJØ bE "Purity/2FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPÓ£M¤;Ù b6 jet/accuracy_7_of_7"trainer/global_step2:JP'HÚ bC Purity/1FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JPŒCT½HÛ bC Purity/3FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPR|JÜ bE "Purity/1FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPÀÆNÝ bI &Purity/1FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP0j™Ì@Þ b; particle/accuracy_1_of_1"trainer/global_step2:JP¬h)gHß bC Purity/3FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPÿµHà bC Purity/1FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JP%´ŽHá bC Purity/4FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP ˜!Jâ bE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP|‰mAHã bC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPúDNä bI &Purity/2FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPà£z;å b6 jet/accuracy_3_of_4"trainer/global_step2:JPvV7p@æ b; particle/accuracy_1_of_3"trainer/global_step2:JP.åJç bE "Purity/3FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPùVxJè bE "Purity/2FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÎ?Jé bE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPª3ÐQ@ê b; particle/accuracy_4_of_4"trainer/global_step2:JP ÏÓ7Hë bC Purity/3FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPl½‡T;ì b6 jet/accuracy_4_of_7"trainer/global_step2:JP¨@¹Jí bE "Purity/1FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPXîHJî bE "Purity/*FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP¶’Hü bC Purity/*FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JP $Hý bC Purity/2FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPÈï˃Jþ bE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP¡<\ÿJÿ bE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP°õ¤N€ bI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPêÁ8J bE "Purity/0FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPÓWJ‚ bE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¾Ž7íNƒ bI &Purity/2FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP,U ‰N„ bI &Purity/4FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPÝäéN… bI &Purity/4FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP¿•™DJ† bE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP ‚cJ‡ bE "Purity/1FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPAŸÆJˆ bE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP´¨ýJ‰ bE "Purity/0FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP¦yãJŠ bE "Purity/0FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPQûÈH‹ bC Purity/*FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JP ©åÞJŒ bE "Purity/1FRt3SRqqt2FBt/event_purity"trainer/global_step2:JP »º H bC Purity/2FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JPC’°ÞHŽ bC Purity/*FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPzѹ¤N bI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP†h N bI &Purity/4FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPN'=‡J‘ bE "Purity/0FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPÓàÐ^H’ bC Purity/1FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPú¢’J“ bE "Purity/0FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP{ê‡òJ” bE "Purity/1FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPŒÂšuJ• bE "Purity/2FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPð/cH– bC Purity/*FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPLÀ¶ˆ;— b6 loss/val_total_loss"trainer/global_step2:JPóEÀN˜ bI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPûsZN™ bI &Purity/4FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPë…™DJš bE "Purity/3FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP!cëJ› bE "Purity/3FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¶slHœ bC Purity/*FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JPäPšIN bI &Purity/2FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP“HD‘Nž bI &Purity/3FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPªÀœNŸ bI &Purity/4FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP$rÚ´@  b; particle/accuracy_2_of_6"trainer/global_step2:JP²ko¶H¡ bC Purity/1FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JPãGZ&H¢ bC Purity/*FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JP±‘ѬJ£ bE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP?XɨN¤ bI &Purity/3FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPñxLLJ¥ bE "Purity/0FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP51ï;¦ b6 jet/accuracy_2_of_6"trainer/global_step2:JP]€ÖJ§ bE "Purity/4FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¬VÏH¨ bC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPv0ëáJ© bE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPvtÌNª bI &Purity/2FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPX"8J« bE "Purity/3FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¥ê»ÇH¬ bC Purity/1FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JPDM²eH­ bC Purity/4FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP'ÌøN® bI &Purity/0FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP¶q?J¯ bE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPu8òJ° bE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP{6J± bE "Purity/3FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÛi N² bI &Purity/4FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPYÕXõH³ bC Purity/1FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP>_H´ bC Purity/*FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPŠ¼Ë!Jµ bE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPɪ¨fH¶ bC Purity/*FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JP¨A1J· bE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP4bÁýJ¸ bE "Purity/2FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPf¼N¹ bI &Purity/2FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPá\~4Nº bI &Purity/4FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP~NÌöH» bC Purity/*FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JPƒ +|J¼ bE "Purity/2FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP¹^UêJ½ bE "Purity/0FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPi:ù%N¾ bI &Purity/1FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPF–RN¿ bI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP@ucNÀ bI &Purity/1FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPVÌ\lJÁ bE "Purity/1FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPb(ã-N bI &Purity/4FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP2߀Hà bC Purity/*FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JPgªÎYJÄ bE "Purity/*FRt3SRqqt2FBt/event_purity"trainer/global_step2:JP=ˆAHâ bC Purity/1FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPÆÈ?Jã bE "Purity/4FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP*–Ð,Nä bI &Purity/*FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP–\nHå bC Purity/1FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPáÜáHæ bC Purity/1FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPS‹Jç bE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP9—vVNè bI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP'ê–Né bI &Purity/*FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPy‚6¿Hê bC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP¹§Hë bC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPѲ¼wJì bE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPaMmNí bI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPaVj@î b; particle/accuracy_3_of_5"trainer/global_step2:JP£×®Hï bC Purity/3FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPìfÍJð bE "Purity/*FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPkJñ bE "Purity/2FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPúú‘Nò bI &Purity/4FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPÀ™‡{;ó b6 jet/accuracy_5_of_5"trainer/global_step2:JPî½qHô bC Purity/*FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JPÔŽNõ bI &Purity/*FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPÌ×ýÎJö bE "Purity/*FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPJjTJ÷ bE "Purity/*FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP™+ØNø bI &Purity/4FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP¶&ÝŸJù bE "Purity/4FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPªñáNú bI &Purity/*FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP-%]ªHû bC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP`c£¬Jü bE "Purity/*FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP•î¾Jý bE "Purity/0FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPÍùá~Jþ bE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP5ø Hÿ bC Purity/2FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPQC!H€ bC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP°H bC Purity/2FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPâ”_H‚ bC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPXì‡Nƒ bI &Purity/*FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP8}Í;„ b6 jet/accuracy_1_of_2"trainer/global_step2:JP#G£;… b6 jet/accuracy_1_of_3"trainer/global_step2:JP3¾ÏH† bC Purity/3FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPö‹H‡ bC Purity/*FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JPá‰ò•Hˆ bC Purity/*FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JPËGoJ‰ bE "Purity/1FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPcRúNŠ bI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP²¿÷J‹ bE "Purity/*FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPòVhHŒ bC Purity/1FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JPÜî¼±H® bC Purity/3FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPN~¤Ë@¯ b; particle/accuracy_2_of_4"trainer/global_step2:JPB †H° bC Purity/2FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JPmŽþxJ± bE "Purity/1FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP¤‹ H² bC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPCµÌêJ³ bE "Purity/0FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP[0H´ bC Purity/1FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JP?=¯EJµ bE "Purity/1FRt1SRqqt4FBt/event_purity"trainer/global_step2:JP%úJ¶ bE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPødÔ>J· bE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPøâô6N¸ bI &Purity/3FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPÿMñG;¹ b6 jet/accuracy_1_of_6"trainer/global_step2:JPò}‚ Hº bC Purity/0FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JPªVþÔJ» bE "Purity/*FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPnK¤H¼ bC Purity/1FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JP¶ûèÛN½ bI &Purity/3FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP F Ç;¾ b6 jet/accuracy_4_of_6"trainer/global_step2:JP8UJ¿ bE "Purity/*FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPð¹aHÀ bC Purity/0FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPi7£ôHÁ bC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPÌ+¶4N bI &Purity/2FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPÑú­§Jà bE "Purity/*FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP´À8®;Ä b6 jet/accuracy_2_of_7"trainer/global_step2:JPŽÄvª;Å b6 jet/accuracy_3_of_7"trainer/global_step2:JPî;IHÆ bC Purity/1FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JPëšçˆJÇ bE "Purity/0FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP5ΔHÈ bC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP`ÂHÉ bC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPèkŒJNÊ bI &Purity/0FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPâønHNË bI &Purity/1FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPŒbNÌ bI &Purity/4FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP¥;HÍ bC Purity/1FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JPy‹™ÃJÎ bE "Purity/*FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPdÂrHÏ bC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP²$2TJÐ bE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPñ«˜ÚJÑ bE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP£Žþ@Ò b; particle/accuracy_2_of_5"trainer/global_step2:JPd“JÓ bE "Purity/0FRt4SRqqt1FBt/event_purity"trainer/global_step2:JP G5JÔ bE "Purity/4FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPçVð2Õ - _wandbruntime‚1407  _runtime‚1407uZ—ÐXÖ ‹¥ ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5857638716697693 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 particle/accuracy_5_of_7‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 +particle/specificity‚0.9371722936630249 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4166666567325592 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 particle/accuracy_3_of_7‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.48038244247436523 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9877232313156128 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.21246246993541718 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.6875 ) Purity/*FRt4SRqqt0FBt/FRt_purity‚0.25 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.0869656577706337 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1751343458890915 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3638758957386017 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12121212482452393 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.30066952109336853 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9991645812988281 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5912697911262512 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 particle/accuracy_6_of_6‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.27341240644454956 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.946180522441864 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2742637097835541 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16970443725585938 particle/accuracy_2_of_2‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.0714285746216774 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 *jet/accuracy_2_of_2‚0.0842399075627327 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16916993260383606 ,jet/accuracy_3_of_3‚0.019832363352179527 *jet/accuracy_1_of_4‚0.8362983465194702 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6499029397964478 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.978643000125885 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9982202649116516 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34062495827674866 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.25568458437919617 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.29735273122787476 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4073259234428406 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4004629850387573 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.4000000059604645 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17556509375572205 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3459821045398712 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.27500003576278687 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.16708260774612427 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.20000000298023224 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.25925925374031067 _step‚98 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.010360523127019405 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14888890087604523 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 6 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.150894433259964 +jet/accuracy_3_of_5‚0.04343318194150925 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9259259700775146 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4956934452056885 particle/accuracy_1_of_4‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929893612861633 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08865249156951904 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5490323305130005 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.40625 jet/accuracy_6_of_7‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5310960412025452 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.960488498210907 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2036145031452179 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.35123324394226074 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.1875 particle/accuracy_2_of_3‚1 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3580247163772583 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_7_of_7‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5763679146766663 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3135964870452881 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,jet/accuracy_3_of_4‚0.046296536922454834 particle/accuracy_1_of_3‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6343722939491272 particle/accuracy_4_of_4‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.20000000298023224 jet/accuracy_4_of_7‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.34892168641090393 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.34732726216316223 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9851412773132324 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.22431346774101257 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5035538673400879 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.203929603099823 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1826390027999878 +jet/accuracy_2_of_5‚0.38879290223121643 (particle/accuracy‚0.8321241736412048 ) Purity/*FRt4SRqqt*FBt/FRt_purity‚0.25 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5726510286331177 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.012696171179413795 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.026280324906110764 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.37713512778282166 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.2934701442718506 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2311890721321106 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.006676744669675827 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.06140350550413132 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.23278865218162537 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.898148238658905 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0729166641831398 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.8888888955116272 *loss/val_total_loss‚52.650225823742446 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4166666567325592 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08038752526044846 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.5512820482254028 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2065751999616623 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3317377269268036 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5107190608978271 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3584112226963043 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.30571261048316956 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703284263611 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.10070713609457016 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.16553793847560883 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ) Purity/*FRt*SRqqt4FBt/FBt_purity‚0.75 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.51481032371521 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ) Purity/*FRt0SRqqt4FBt/FBt_purity‚0.75 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3647727966308594 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.983426570892334 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 ,particle/sensitivity‚0.26444312930107117 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11537210643291473 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5240161418914795 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.24591922760009766 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4042842388153076 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4518648684024811 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9270833134651184 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.295330286026001 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19633832573890686 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9951723217964172 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.2607461214065552 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 particle/accuracy_3_of_5‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.1868423968553543 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.011507936753332615 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.19975726306438446 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0992063581943512 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8135844469070435 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1328887641429901 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.17432431876659393 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19248026609420776 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_1_of_2‚0.6358675956726074 *jet/accuracy_1_of_3‚0.7684093713760376 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.31079405546188354 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9825581312179565 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13577282428741455 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13963761925697327 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5615400075912476 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 particle/accuracy_6_of_7‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9820717573165894 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.40229886770248413 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.35185185074806213 )jet/accuracy_2_of_3‚0.191656693816185 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5050316452980042 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_6‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.45459768176078796 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_6‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.12643678486347198 particle/accuracy_2_of_4‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9854693412780762 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.23976030945777893 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.2813228666782379 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3201262354850769 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_6‚0.9807692170143127 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.10243802517652512 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.15964913368225098 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_4_of_6‚0.01923076994717121 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2004161924123764 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1944444477558136 jet/accuracy_2_of_7‚0 jet/accuracy_3_of_7‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3205467462539673 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9906035661697388 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9982202649116516 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.26667657494544983 particle/accuracy_2_of_5‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.44126981496810913 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3911451995372772 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05395380035042763 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9856889247894287 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt0SRqqt1FBt/FXüôÂ9Rt_purity‚0.29367563128471375 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1842220574617386 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.17664584517478943 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9771882891654968 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.36683186888694763 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.12009488791227341 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6025640964508057 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.17594720423221588 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1878787875175476 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1437658965587616 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9736842513084412 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.09150204062461853 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.005803571082651615 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2210361510515213 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9920634627342224 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4307277202606201 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.27284735441207886 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9628840088844299 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 'particle/f_score‚0.3296707570552826 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15760786831378937 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.024635178968310356 particle/accuracy_1_of_6‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 epoch‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9937106966972351 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.003427128540351987 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.8571428656578064 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4848548471927643 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6666666269302368 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15189866721630096 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9922024011611938 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.029579931870102882 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.018265200778841972 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.29964175820350647 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.8871528506278992 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.008185812272131443 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5458333492279053 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2172619104385376 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.16036172211170197 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.038382597267627716 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9842718243598938 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.1708754450082779 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.2877906858921051 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.3851441740989685 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.18936781585216522 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2636972963809967 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.30745750665664673 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.29924243688583374 ,jet/accuracy_4_of_5‚0.004480286967009306 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_7‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9906461834907532 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 *jet/accuracy_1_of_5‚0.8582502007484436 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4374999701976776 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3576388955116272 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1493547260761261 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.31612807512283325 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9655850529670715 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.12126712501049042 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 particle/accuracy_7_of_7‚1 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.28859570622444153 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.14827586710453033 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 -jet/accuracy_4_of_4‚0.0021919880528002977 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.32852116227149963 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.111111119389534 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6842105984687805 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 % _timestamp‚1.7884973498990288e+09 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.23040622472763062 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.006601158995181322 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8698830604553223 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.2790291905403137 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9767857193946838 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3260684609413147 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_7‚1 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9858832359313965 particle/accuracy_4_of_6‚1 particle/accuracy_5_of_6‚1 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.13497787714004517 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.002713336143642664 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.012544036842882633 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_3‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9859945178031921 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9875328540802002 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.25568458437919617 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_5_of_5‚1 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.6875 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.8928571343421936 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9851812124252319 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.18981483578681946 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1944444477558136 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1328514814376831 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 trainer/global_step‚1208 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.11964781582355499 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.3229745626449585 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4699999988079071 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 _runtime‚1407.678512648 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0030261483043432236 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.11696784198284149 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.02916516549885273 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.2866666913032532 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3228037357330322 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9989227056503296 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9846491813659668 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6969696879386902 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971829056739807 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.29920709133148193 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15860912203788757 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25808969140052795 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.125 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.06525366753339767 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9300000071525574 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.38248148560523987 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2982129752635956 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9953256845474243 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.29890596866607666 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1919122189283371 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.12126712501049042 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3409891426563263 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1357496827840805 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.25340038537979126 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9863973259925842 +jet/accuracy_3_of_6‚0.13461539149284363 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25ékk0F× ¥ 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9854693412780762 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.23976030945777893 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.2813228666782379 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3201262354850769 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_6‚0.9807692170143127 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.10243802517652512 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.15964913368225098 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_4_of_6‚0.01923076994717121 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2004161924123764 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1944444477558136 jet/accuracy_2_of_7‚0 jet/accuracy_3_of_7‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3205467462539673 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9906035661697388 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9982202649116516 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.26667657494544983 particle/accuracy_2_of_5‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.44126981496810913 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3911451995372772 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05395380035042763 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9856889247894287 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.29367563128471375 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1842220574617386 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.17664584517478943 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9771882891654968 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.36683186888694763 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.12009488791227341 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6025640964508057 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.17594720423221588 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1878787875175476 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1437658965587616 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9736842513084412 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.09150204062461853 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.005803571082651615 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2210361510515213 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9920634627342224 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4307277202606201 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.27284735441207886 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9628840088844299 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 'particle/f_score‚0.3296707570552826 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15760786831378937 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.024635178968310356 particle/accuracy_1_of_6‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 epoch‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9937106966972351 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.003427128540351987 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.8571428656578064 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4848548471927643 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6666666269302368 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15189866721630096 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9922024011611938 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.029579931870102882 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.018265200778841972 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.29964175820350647 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.8871528506278992 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.008185812272131443 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5458333492279053 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2172619104385376 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.16036172211170197 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.038382597267627716 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9842718243598938 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.1708754450082779 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.2877906858921051 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.3851441740989685 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.18936781585216522 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2636972963809967 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.30745750665664673 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.29924243688583374 ,jet/accuracy_4_of_5‚0.004480286967009306 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_7‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9906461834907532 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 *jet/accuracy_1_of_5‚0.8582502007484436 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4374999701976776 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3576388955116272 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1493547260761261 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.31612807512283325 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9655850529670715 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.12126712501049042 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 particle/accuracy_7_of_7‚1 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.28859570622444153 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.14827586710453033 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 -jet/accuracy_4_of_4‚0.0021919880528002977 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.32852116227149963 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.111111119389534 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6842105984687805 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 % _timestamp‚1.7884973498990288e+09 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.23040622472763062 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.006601158995181322 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8698830604553223 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.2790291905403137 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9767857193946838 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3260684609413147 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_7‚1 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9858832359313965 particle/accuracy_4_of_6‚1 particle/accuracy_5_of_6‚1 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.13497787714004517 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.002713336143642664 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.012544036842882633 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_3‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9859945178031921 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9875328540802002 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.25568458437919617 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_5_of_5‚1 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.6875 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.8928571343421936 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9851812124252319 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.18981483578681946 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1944444477558136 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1328514814376831 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 trainer/global_step‚1208 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.11964781582355499 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.3229745626449585 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4699999988079071 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 _runtime‚1407.678512648 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0030261483043432236 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.11696784198284149 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.02916516549885273 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.2866666913032532 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3228037357330322 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9989227056503296 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9846491813659668 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6969696879386902 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971829056739807 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.29920709133148193 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15860912203788757 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25808969140052795 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.125 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.06525366753339767 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9300000071525574 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.38248148560523987 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2982129752635956 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9953256845474243 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.29890596866607666 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1919122189283371 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.12126712501049042 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3409891426563263 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1357496827840805 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.25340038537979126 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9863973259925842 +jet/accuracy_3_of_6‚0.13461539149284363 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5857638716697693 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 particle/accuracy_5_of_7‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/eveÅZ!JfLnt_proportion‚0.015376984141767025 +particle/specificity‚0.9371722936630249 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4166666567325592 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 particle/accuracy_3_of_7‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.48038244247436523 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9877232313156128 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.21246246993541718 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.6875 ) Purity/*FRt4SRqqt0FBt/FRt_purity‚0.25 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.0869656577706337 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1751343458890915 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3638758957386017 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12121212482452393 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.30066952109336853 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9991645812988281 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5912697911262512 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 particle/accuracy_6_of_6‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.27341240644454956 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.946180522441864 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2742637097835541 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16970443725585938 particle/accuracy_2_of_2‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.0714285746216774 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 *jet/accuracy_2_of_2‚0.0842399075627327 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16916993260383606 ,jet/accuracy_3_of_3‚0.019832363352179527 *jet/accuracy_1_of_4‚0.8362983465194702 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6499029397964478 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.978643000125885 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9982202649116516 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34062495827674866 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.25568458437919617 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.29735273122787476 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4073259234428406 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4004629850387573 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.4000000059604645 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17556509375572205 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3459821045398712 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.27500003576278687 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.16708260774612427 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.20000000298023224 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.25925925374031067 _step‚98 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.010360523127019405 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14888890087604523 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 6 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.150894433259964 +jet/accuracy_3_of_5‚0.04343318194150925 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9259259700775146 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4956934452056885 particle/accuracy_1_of_4‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929893612861633 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08865249156951904 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5490323305130005 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.40625 jet/accuracy_6_of_7‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5310960412025452 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.960488498210907 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2036145031452179 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.35123324394226074 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.1875 particle/accuracy_2_of_3‚1 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3580247163772583 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_7_of_7‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5763679146766663 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3135964870452881 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,jet/accuracy_3_of_4‚0.046296536922454834 particle/accuracy_1_of_3‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6343722939491272 particle/accuracy_4_of_4‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.20000000298023224 jet/accuracy_4_of_7‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.34892168641090393 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.34732726216316223 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9851412773132324 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.22431346774101257 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5035538673400879 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.203929603099823 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1826390027999878 +jet/accuracy_2_of_5‚0.38879290223121643 (particle/accuracy‚0.8321241736412048 ) Purity/*FRt4SRqqt*FBt/FRt_purity‚0.25 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5726510286331177 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.012696171179413795 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.026280324906110764 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.37713512778282166 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.2934701442718506 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2311890721321106 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.006676744669675827 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.06140350550413132 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.23278865218162537 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.898148238658905 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0729166641831398 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.8888888955116272 *loss/val_total_loss‚52.650225823742446 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4166666567325592 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08038752526044846 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.5512820482254028 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2065751999616623 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3317377269268036 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5107190608978271 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3584112226963043 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.30571261048316956 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703284263611 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.10070713609457016 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.16553793847560883 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ) Purity/*FRt*SRqqt4FBt/FBt_purity‚0.75 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.51481032371521 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ) Purity/*FRt0SRqqt4FBt/FBt_purity‚0.75 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3647727966308594 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.983426570892334 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 ,particle/sensitivity‚0.26444312930107117 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11537210643291473 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5240161418914795 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.24591922760009766 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4042842388153076 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4518648684024811 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9270833134651184 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.295330286026001 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19633832573890686 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9951723217964172 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.2607461214065552 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 particle/accuracy_3_of_5‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.1868423968553543 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.011507936753332615 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.19975726306438446 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0992063581943512 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8135844469070435 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1328887641429901 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.17432431876659393 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19248026609420776 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_1_of_2‚0.6358675956726074 *jet/accuracy_1_of_3‚0.7684093713760376 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.31079405546188354 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9825581312179565 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13577282428741455 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13963761925697327 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5615400075912476 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 particle/accuracy_6_of_7‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9820717573165894 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.40229886770248413 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.35185185074806213 )jet/accuracy_2_of_3‚0.191656693816185 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5050316452980042 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_6‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.45459768176078796 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_6‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.12643678486347198 particle/accuracy_2_of_4‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75b,<ÀØ ‚(:… È›éÔØn gpu.0.gpu‚5.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚33.0 gpu.0.powerWatts‚86.502' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚21.6255 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åŸC÷]Ù ‚(:Ò È›éÔ‘”ön disk.sda.in‚ 1.3984375 disk.sda1.out‚0 disk.sda2.out‚ 154.66015625 disk.rbd2.in‚0 disk.rbd2.out‚ 0.41650390625 disk.sda.out‚ 154.66015625 disk.sdb2.in‚ 0.23046875 disk.rbd0.in‚0 disk.sdb1.in‚0 proc.cpu.threads‚17 network.recv‚525902" disk.nvme0n1.out‚ 5419.33984375 disk.md0.out‚ 148.73046875 disk.sdb1.out‚0" proc.memory.rssMB‚ 3650.1171875 disk.dm-0.out‚ 14879.921875 disk.nvme1n1.in‚ 9.0078125" disk.nvme1n1.out‚ 9460.58203125 disk.md0.in‚ 1.62890625 disk.sdb2.out‚ 154.66015625* proc.memory.percent‚22.278547286987305 disk.md0p1.in‚ 1.62890625 disk.md0p1.out‚ 148.73046875 disk.dm-0.in‚ 65.56640625 disk.rbd0.out‚ 0.279296875$ memory_percent‚99.98538494110107& proc.memory.availableMB‚ 2.39453125 cpu‚25.219940753174686) disk./.usagePercent‚83.71718876712713 disk.sdb.out‚ 154.66015625 disk.nvme0n1.in‚ 56.55859375 disk.sda1.in‚0 network.sent‚ 1.790192e+06% disk./.usageGB‚2994.1321754455566 disk.sdb.in‚ 0.23046875 disk.sda2.in‚ 1.3984375MìºæUÚ ‚(:Ê ×›éÔÓœâl) disk./.usagePercent‚83.75658532791373 disk.md0.out‚ 150.265625& proc.memory.availableMB‚ 1322.21875# proc.memory.rssMB‚ 2992.30859375 cpu‚25.217916990906076 network.sent‚ 1.915264e+06 disk.md0.in‚ 1.62890625 disk.sda1.in‚0* proc.memory.percent‚18.263602256774902 disk.rbd2.out‚ 0.41650390625 disk.rbd0.in‚0 disk.md0p1.out‚ 150.265625 disk.dm-0.out‚15729.66015625 disk.nvme0n1.in‚ 56.55859375$ memory_percent‚91.92981719970703 proc.cpu.threads‚17 disk.sda.in‚ 1.3984375 network.recv‚531623 disk.dm-0.in‚ 65.56640625 disk.nvme1n1.in‚ 9.0078125 disk.sdb2.out‚ 156.2578125 disk.sda1.out‚0 disk.md0p1.in‚ 1.62890625 disk.sdb.in‚ 0.23046875 disk.sda2.out‚ 156.2578125 disk.nvme1n1.out‚ 9618.390625 disk.sda.out‚ 156.2578125$ disk./.usageGB‚2995.541187286377 disk.sdb1.out‚0 disk.sda2.in‚ 1.3984375" disk.nvme0n1.out‚ 6111.26953125 disk.rbd0.out‚ 0.279296875 disk.sdb1.in‚0 disk.sdb.out‚ 156.2578125 disk.rbd2.in‚0 disk.sdb2.in‚ 0.23046875 ûÍ“Û ‚(:ˆ ×›éÔ»ñl gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.283' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.07075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D«Ö“Ü ‚(:ˆ æ›éÔï”…m gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚163.597' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.89925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ^q_dÝ ‚(:Ù æ›éÔ“ ªm! disk.sda2.out‚157.89111328125 disk.md0.in‚ 1.62890625$ memory_percent‚91.93212985992432 disk.sda1.out‚0 network.sent‚ 1.921523e+06 network.recv‚536531 disk.md0.out‚ 151.828125 disk.sdb2.in‚ 0.23046875 disk.sda.out‚157.89111328125) proc.memory.availableMB‚ 1321.83984375) disk./.usagePercent‚83.76000081147332 disk.rbd2.in‚0! disk.sdb2.out‚157.89111328125 disk.dm-0.out‚ 16588.5390625 disk.nvme1n1.in‚ 9.0078125 disk.rbd0.out‚ 0.279296875* proc.memory.percent‚18.263602256774902 cpu‚25.21549579043534 disk.md0p1.out‚ 151.828125 disk.sdb1.out‚0 disk.md0p1.in‚ 1.62890625$ disk./.usageGB‚2995.663341522217 disk.sda2.in‚ 1.3984375! disk.nvme0n1.out‚ 6865.1796875 disk.sda1.in‚0 disk.dm-0.in‚ 65.56640625 disk.sdb1.in‚0 disk.sdb.in‚ 0.23046875 disk.sdb.out‚157.89111328125 disk.nvme0n1.in‚ 56.55859375 disk.sda.in‚ 1.3984375 proc.cpu.threads‚17 disk.nvme1n1.out‚ 9723.359375 disk.rbd2.out‚ 0.4404296875 disk.rbd0.in‚0# proc.memory.rssMB‚ 2992.30859375s´9“Þ ‚(:ˆ õ›éÔ´Ìòl gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.461' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.11525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍÃ@_ß ‚(:Ô õ›éԵɫm disk.sda2.in‚ 1.3984375 disk.nvme0n1.in‚ 56.55859375 disk.md0.in‚ 1.62890625$ memory_percent‚91.93344116210938$ proc.memory.availableMB‚1321.625 disk.sdb2.out‚ 159.857421875 disk.sdb1.in‚0# proc.memory.rssMB‚ 2992.31640625 disk.sda.out‚ 159.857421875 cpu‚25.213311394914747 disk.md0p1.in‚ 1.62890625 disk.sda2.out‚ 159.857421875 disk.nvme1n1.in‚ 9.0078125* proc.memory.percent‚18.263649940490723 disk.sdb.in‚ 0.23046875 disk.md0p1.out‚ 153.72265625! disk.nvme0n1.out‚ 7028.0390625) disk./.usagePercent‚83.76699091610645$ disk./.usageGB‚2995.913341522217" disk.nvme1n1.out‚ 9828.33203125 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 65.56640625 disk.dm-0.out‚16856.37109375 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd2.out‚ 0.4404296875 disk.sdb2.in‚ 0.23046875 disk.sda.in‚ 1.3984375 disk.sda1.out‚0 disk.rbd0.out‚ 0.279296875 disk.md0.out‚ 153.72265625 network.sent‚ 1.926817e+06 disk.sdb.out‚ 159.857421875 disk.sda1.in‚0 network.recv‚5399808 ‚2à - _wandbruntime‚1460  _runtime‚1460£Æá À _step‚99 & lr-AdamW/pg1‚0.0010523186457272345 & lr-AdamW/pg2‚0.0010523186457272345 trainer/global_step‚1249 % _timestamp‚1.7884974030836022e+09 _runtime‚1460.860887685ZÇÊâ Ä _step‚99 & lr-AdamW/pg1‚0.0010523186457272345 & lr-AdamW/pg2‚0.0010523186457272345 trainer/global_step‚1249 % _timestamp‚1.7884974030836022e+09 _runtime‚1460.860887685cLú’È2ã - _wandbruntime‚1460  _runtime‚1460¹®Ï¼†ä æO G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _runtime‚1460.861799569 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57855045 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚106899757519872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚106898829546496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15690706 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚1249 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26185782 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26185782 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1433606296064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 _step‚100 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31695607 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116347976886784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚103398936096349 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚103398011081993 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1427338659577 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105466151223808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57881389 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7691014 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1433606296064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26185780 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚101970672422416 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 % _timestamp‚1.7884974030843995e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105466133220864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31669265 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23381720 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1502338469888 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7690403 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116348000826880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1432696325632 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚105466151223808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228783104 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_balŸ ¹ftch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26185780 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252723200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1432696325632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15690704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚106899757519872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23381107 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚117850086573568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚611 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚106898829546496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105466133220864 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚101970689461776 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1428246634573 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23381107 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚117850086573568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚611 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚106898829546496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105466133220864 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚101970689461776 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1428246634573 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15690706 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚1249 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26185782 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26185782 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1433606296064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 _step‚100 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31695607 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116347976886784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚103398936096349 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚103398011081993 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1427338659577 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105466151223808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57881389 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7691014 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1433606296064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26185780 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚101970672422416 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 % _timestamp‚1.7884974030843995e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105466133220864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31669265 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23381720 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1502338469888 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7690403 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116348000826880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0dhY2æ - _wandbruntime‚1461  _runtime‚1461+Hºbõ&ç ïM P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚105547752306688 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57926406 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1434727769600 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7696725 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1428455932855 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23399496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15702769 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚105547752306688 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117941719178240 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚102049707056320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1429363908059 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15702771 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57926406 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31720464 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23398890 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _runtime‚1461.939860018 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31694096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1503492646912 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚103479070964379 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1503251294208 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57900036 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1433817785856 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚106982498079232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 265292800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚604 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7696121 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1433817785856 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚102049690016960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚105547770309632 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105547770309632 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26205940 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚116438226531328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26205942 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚106982498079232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚116438202591232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚106981570092544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31694096 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784uH6Ç`"è óM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚102049690016960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚105547770309632 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105547770309632 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26205940 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚116438226531328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26205942 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚106982498079232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚116438202591232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚106981570092544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31694096 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚105547752306688 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57926406 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1434727769600 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7696725 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1428455932855 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23399496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15702769 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚105547752306688 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117941719178240 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚102049707056320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1429363908059 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15702771 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57926406 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31720464 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23398890 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _runtime‚1461.939860018 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31694096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1503492646912 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚103479070964379 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1503251294208 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 C6DeviceStatsMonitor.on_train_ë™·-™batch_end/active.all.freed‚57900036 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1433817785856 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚106982498079232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 265292800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚604 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7696121 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1433817785856 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570eåÄÔ¾2é - _wandbruntime‚1461  _runtime‚1461QñW/Yê Ó /loss/FRt4/assignment_loss‚5.275053465807879 2loss/SRqqt2/assignment_loss‚2.1623462639950417 /loss/FRt4/detection_loss‚2.7881158921453686 /loss/SRqqt1/detection_loss‚2.12172112793758 _runtime‚1461.941729786 2loss/SRqqt3/assignment_loss‚1.5041565852701118 1loss/FBt3/assignment_loss‚0.32028522580575164 /loss/FBt1/detection_loss‚1.3319798456646654 # _timestamp‚1.78849740416416e+09 /loss/FRt2/assignment_loss‚3.649939882344213 2loss/SRqqt1/assignment_loss‚1.8456280745308975 2loss/SRqqt4/assignment_loss‚2.3385388595717296 1loss/FBt1/assignment_loss‚0.18670139692097115 0loss/FBt4/assignment_loss‚0.8601522435298344 0loss/FRt3/assignment_loss‚2.4025788023823598 0loss/FBt2/assignment_loss‚0.5592140946537256 %loss/total_loss‚53.26269627023168 .loss/FRt1/detection_loss‚1.775641436301745 /loss/FRt3/detection_loss‚1.9893806256720277 /loss/FBt4/detection_loss‚3.7016559143861136 0loss/FRt1/assignment_loss‚2.9380365894390987 /loss/FBt3/detection_loss‚3.6716696534837996 epoch‚1 trainer/global_step‚1249 /loss/FRt2/detection_loss‚2.7806251562874893 1loss/SRqqt2/detection_loss‚2.1494585694493473 1loss/SRqqt3/detection_loss‚1.1430045818790384 _step‚102 1loss/SRqqt4/detection_loss‚3.0238182573091414 .loss/FBt2/detection_loss‚2.742993725463748•F¹ü]ë × .loss/FRt1/detection_loss‚1.775641436301745 /loss/FRt3/detection_loss‚1.9893806256720277 /loss/FBt4/detection_loss‚3.7016559143861136 0loss/FRt1/assignment_loss‚2.9380365894390987 /loss/FBt3/detection_loss‚3.6716696534837996 epoch‚1 trainer/global_step‚1249 /loss/FRt2/detection_loss‚2.7806251562874893 1loss/SRqqt2/detection_loss‚2.1494585694493473 1loss/SRqqt3/detection_loss‚1.1430045818790384 _step‚102 1loss/SRqqt4/detection_loss‚3.0238182573091414 .loss/FBt2/detection_loss‚2.742993725463748 /loss/FRt4/assignment_loss‚5.275053465807879 2loss/SRqqt2/assignment_loss‚2.1623462639950417 /loss/FRt4/detection_loss‚2.7881158921453686 /loss/SRqqt1/detection_loss‚2.12172112793758 _runtime‚1461.941729786 2loss/SRqqt3/assignment_loss‚1.5041565852701118 1loss/FBt3/assignment_loss‚0.32028522580575164 /loss/FBt1/detection_loss‚1.3319798456646654 # _timestamp‚1.78849740416416e+09 /loss/FRt2/assignment_loss‚3.649939882344213 2loss/SRqqt1/assignment_loss‚1.8456280745308975 2loss/SRqqt4/assignment_loss‚2.3385388595717296 1loss/FBt1/assignment_loss‚0.18670139692097115 0loss/FBt4/assignment_loss‚0.8601522435298344 0loss/FRt3/assignment_loss‚2.4025788023823598 0loss/FBt2/assignment_loss‚0.5592140946537256 %loss/total_loss‚53.26269627023168fØÎS²‘ì ‚(:† „œéÔ ­–n gpu.0.gpu‚34.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.992' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”ï”3gí ‚(:Ü „œéÔÀ©‹o disk.sda2.in‚ 1.3984375 disk.nvme1n1.out‚ 10038.09375) disk./.usagePercent‚83.76700371537031# proc.memory.rssMB‚ 2992.31640625 proc.cpu.threads‚17 disk.md0p1.out‚ 155.38671875 disk.sdb.out‚161.5888671875 disk.md0.in‚ 1.62890625 disk.rbd2.in‚0 disk.rbd2.out‚ 0.50341796875 disk.sdb2.in‚ 0.23046875( proc.memory.availableMB‚ 1320.6328125* proc.memory.percent‚18.263649940490723 network.sent‚ 1.93055e+06 disk.md0p1.in‚ 1.62890625 disk.sdb.in‚ 0.23046875 disk.nvme0n1.in‚ 56.55859375" disk.nvme0n1.out‚ 7125.69140625 disk.sda.in‚ 1.3984375$ memory_percent‚91.93949699401855 network.recv‚544274 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-0.out‚17163.78515625 disk.sdb2.out‚161.5888671875 disk.rbd0.out‚ 0.279296875 disk.sda.out‚161.5888671875 cpu‚25.211110060789817 disk.sdb1.in‚0 disk.sda2.out‚161.5888671875 disk.md0.out‚ 155.38671875% disk./.usageGB‚2995.9137992858887 disk.sda1.out‚0 disk.dm-0.in‚ 65.56640625 disk.nvme1n1.in‚ 9.0078125 disk.sdb1.out‚0‡†L‘î ‚(:† “œéÔÇê†m gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚177.508' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.377 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û¬¯nï ‚(:ã “œéÔ¯…Ým$ disk./.usageGB‚2994.859664916992 disk.sdb.out‚163.01806640625" disk.nvme1n1.out‚ 10126.6015625 disk.sdb2.in‚ 0.23046875 disk.sda.in‚ 1.3984375 disk.sda.out‚163.01806640625# proc.memory.rssMB‚ 2992.31640625* proc.memory.percent‚18.263649940490723 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.md0p1.in‚ 1.62890625 disk.md0p1.out‚ 156.76171875! disk.sda2.out‚163.01806640625$ memory_percent‚91.94610118865967 network.sent‚ 2.037831e+06) disk./.usagePercent‚83.73752967722645 disk.sdb.in‚ 0.23046875 disk.md0.out‚ 156.76171875 disk.sda1.out‚0 disk.rbd0.out‚ 0.279296875 disk.nvme0n1.in‚ 56.55859375 disk.nvme1n1.in‚ 9.0078125 disk.rbd2.out‚ 0.50341796875" disk.nvme0n1.out‚ 7223.52734375 cpu‚25.209226202577806 proc.cpu.threads‚17 disk.dm-0.out‚17350.12890625 disk.sda2.in‚ 1.3984375 disk.sdb1.out‚0 disk.dm-0.in‚ 65.56640625 disk.rbd2.in‚0) proc.memory.availableMB‚ 1319.55078125 network.recv‚549614 disk.md0.in‚ 1.62890625! disk.sdb2.out‚163.01806640625 disk.sda1.in‚0E9j“ð ‚(:ˆ ¢œéÔ¤œ¢m gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚182.981' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.74525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M¥¯Ø_ñ ‚(:Ô ¢œéÔÅêÛm disk.nvme1n1.out‚ 10126.71875 network.sent‚ 2.046307e+06 disk.sdb2.out‚164.9814453125 disk.rbd0.out‚ 0.279296875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda2.in‚ 1.3984375 disk.rbd2.out‚ 0.56640625$ memory_percent‚91.94927215576172 proc.cpu.threads‚17 disk.sdb2.in‚ 0.23046875 disk.sda1.out‚0 disk.md0p1.out‚ 158.65234375! disk.nvme0n1.out‚ 7321.1796875# proc.memory.rssMB‚ 2992.31640625 disk.rbd0.in‚0 disk.dm-0.in‚ 65.56640625 disk.sdb.out‚164.9814453125 disk.sda2.out‚164.9814453125 disk.md0.out‚ 158.65234375 disk.rbd2.in‚0 cpu‚25.20716910272991 disk.dm-0.out‚ 17447.8984375 disk.nvme0n1.in‚ 56.55859375 disk.sda.in‚ 1.3984375 disk.sda.out‚164.9814453125 disk.sdb.in‚ 0.23046875& proc.memory.availableMB‚ 1319.03125 network.recv‚554639 disk.md0p1.in‚ 1.62890625 disk.nvme1n1.in‚ 9.0078125 disk.md0.in‚ 1.62890625* proc.memory.percent‚18.263649940490723) disk./.usagePercent‚83.72271420933316$ disk./.usageGB‚2994.329792022705 disk.sda1.in‚0: ™Ù2ò - _wandbruntime‚1513  _runtime‚1513Ä,Æó À & lr-AdamW/pg1‚0.0010521761020453005 & lr-AdamW/pg2‚0.0010521761020453005 trainer/global_step‚1299 $ _timestamp‚1.788497456107392e+09 _runtime‚1513.884710554 _step‚103é]ZÍÊô Ä $ _timestamp‚1.788497456107392e+09 _runtime‚1513.884710554 _step‚103 & lr-AdamW/pg1‚0.0010521761020453005 & lr-AdamW/pg2‚0.0010521761020453005 trainer/global_step‚1299gÁÊóÃ2õ - _wandbruntime‚1513  _runtime‚1513¾›Žì'ö æO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚120859286049280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚111036799236096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 % _timestamp‚1.7884974561086414e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚111035871262720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24268631 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16293956 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27193780 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60107947 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7975303 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚109547087512064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚109547087512064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111035871262720 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚122419457779712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚105921552149616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _runtime‚1513.886461792 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚111036799236096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24269259 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32940509 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1489693721088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16293954 _step‚104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1488783750656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32914167 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚107404768064301 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1484123889681 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32914167 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60134291 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚122419714697216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27193782 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7975303 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚109547087512064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚109547087512064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111035871262720 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚122419457779712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚105921552149616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _runtime‚1513.886461792 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚111036799236096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24269259 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32940509 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1489693721088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16293954 _step‚104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1488783750656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32914167 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚107404768064301 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1484123889681 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32914167 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60134291 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚122419714697216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27193782 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 I2ú - _wandbruntime‚1515  _runtime‚15151鲟"û ïM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60152938 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚111118611808768 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚109628706597888 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120949487813632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109628706597888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1489905210880 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254807040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230866944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32965366 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60152938 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60179308 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32938998 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚106000586783520 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 % _timestamp‚1.7884974572387123e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1490815194624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1484333187963 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24287047 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚122510846159360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1485241163167 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32938998 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24286424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16306019 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚107485827946687 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 _step‚105 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27213942 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚122511100966400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1489905210880 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27213940 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚111119539795456 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27213942 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7981026 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7980405 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32938998 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24286424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16306019 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚107485827946687 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 _step‚105 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27213942 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚122511100966400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1489905210880 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27213940 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚111119539795456 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27213942 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7981026 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7980405 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚109628706597888 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120949487813632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109628706597888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1489905210880 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254807040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230866944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32965366 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60152938 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60179308 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32938998 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚106000586783520 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 % _timestamp‚1.7884974572387123e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1490815194624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1484333187963 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24287047 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚122510846159360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1485241163167 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8258864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚115173840952320 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 _runtime‚1565.844402539 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34185411 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1618646919168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259014656 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 trainer/global_step‚1349 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚113628059806208 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62360849 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62387193 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235074560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚125370547331584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1618411844608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚113628041803264 IDevic(¸;áeStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111411525046609 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1540001144789 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1544871175680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16897204 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1545781146112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚606 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28201782 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34159069 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8259468 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚126988959176192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚109872448916176 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115172912978944 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚113628059806208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28201780 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34159069 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1544871175680 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34185411 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚125370571271680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1545781146112z%Bð'‹êO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8258864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚115173840952320 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 _runtime‚1565.844402539 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34185411 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1618646919168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259014656 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 trainer/global_step‚1349 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚113628059806208 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62360849 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62387193 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235074560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚125370547331584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1618411844608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚113628041803264 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111411525046609 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1540001144789 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1544871175680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16897204 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1545781146112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚606 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28201782 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34159069 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8259468 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚126988959176192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚109872448916176 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115172912978944 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚113628059806208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28201780 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34159069 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1544871175680 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34185411 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚125370571271680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1545781146112lV«2Œ- _wandbruntime‚1566  _runtime‚1566bÄ»iö&ðM E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34210268 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1545992635904 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62405840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16909269 _step‚109 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28221942 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34210268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16909271 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25173832 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚125460773036032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62405840 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34183900 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚113709660889088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚127080610296832 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚109951449471360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1540210443071 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚610 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34183900 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28221940 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1541118418275 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚115256581511680 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚127080361781248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1619813320704 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62432210 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 _runtime‚1566.796466645 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚113709678892032 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248515584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚113709678892032 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34210268 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1545992635904 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62405840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16909269 _step‚109 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28221942 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34210268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16909271 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25173832 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚125460773036032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62405840 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34183900 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚113709660889088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚127080610296832 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚109951449471360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1540210443071 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚610 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34183900 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28221940 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1541118418275 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚115256581511680 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚127080361781248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1619813320704 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62432210 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 _runtime‚1566.796466645 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚113709678892032 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248515584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H@DeviceStatsMonitor.on_tr‡§Çuain_batch_end/segment.small_pool.current‚570 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚113709678892032 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119309954695168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29209782 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35403971 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35430313 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1600958600704 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64640095 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263208960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚113823328643376 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1595878399897 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚119309954695168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17500456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚117708996094464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚611 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35403971 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8545035 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚119310882668544 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1601868571136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26046102 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1601868571136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64613751 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚115419207043273 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26045489 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 239268864 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚113823311604016 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚119310882668544 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚117709014097408 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8545646 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1676965100032 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570¯„¯ï'ŸéO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35430313 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚1399 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚129881856494080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 1620.13872675 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29209780 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚613 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚129881832553984 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚131558558385152 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29209780 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚119309954695168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17500456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚117708996094464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚611 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35403971 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8545035 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚119310882668544 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1601868571136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26046102 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1601868571136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64613751 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚115419207043273 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26045489 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 239268864 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚113823311604016 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚119310882668544 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚117709014097408 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8545646 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1676965100032 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29209782 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1600958600704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119309954695168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29209782 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35403971 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35430313 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1600958600704 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64640095 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263208960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚113823328643376 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1595878399897 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚117790633183232 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚119393623227904 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚119392695241216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚129972058258432 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1602080060928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚598 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚113902329198560 @:DeviceStatsMonitor.on_train_bteå`3!atch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _step‚113 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1596087698179 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚117790633183232 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17512521 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26063266 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚113902346237920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚117790615180288 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1596995673383 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8551345 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 % _timestamp‚1.7884975633354745e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚115499341911303 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64658742 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29229940 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64685112 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35455170 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8550747 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35428802 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚117790633183232 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚119393623227904 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚119392695241216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚129972058258432 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1602080060928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚598 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚113902329198560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _step‚113 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1596087698179 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚117790633183232 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17512521 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26063266 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚113902346237920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚117790615180288 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1596995673383 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8551345 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 % _timestamp‚1.7884975633354745e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚115499341911303 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64658742 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29229940 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64685112 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35455170 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8550747 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35428802 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615qžc È2£- _wandbruntime‚1621  _runtime‚1621ˆ@Z¤Ô epoch‚1 0loss/FRt3/assignment_loss‚2.1400843979539097 .loss/FRt2/detection_loss‚2.945626337081194 0loss/SRqqt2/detection_loss‚3.444748419854376 /loss/FBt1/detection_loss‚1.0360009660306515 2loss/SRqqt3/assignment_loss‚1.1176971080348543 0loss/FBt1/assignment_loss‚0.2970232487378353 /loss/FBt3/detection_loss‚2.8831029401885138 % _timestamp‚1.7884975633380601e+09 0loss/FRt4/assignment_loss‚4.6935389041900635 .loss/FBt4/detection_loss‚2.995287024974823 trainer/global_step‚1399 _runtime‚1621.115051063 /loss/FRt2/assignment_loss‚4.400761805474758 2loss/SRqqt1/assignment_loss‚1.8320182644087692 2loss/SRqqt4/assignment_loss‚2.2975025218725205 0loss/FBt2/assignment_loss‚0.2722888867833212 /loss/FRt1/detection_loss‚2.1354327620578593 /loss/FRt1/assignment_loss‚3.139075889370658 2loss/SRqqt2/assignment_loss‚1.8533146729071934 /loss/FRt3/detection_loss‚1.6386001452400878 1loss/SRqqt3/detection_loss‚1.1423399933006453 _step‚114 /loss/FRt4/detection_loss‚2.9780624017995945 1loss/SRqqt4/detection_loss‚2.8344944071769715 &loss/total_loss‚51.153899523260996 .loss/FBt3/assignment_loss‚0.33556479958093 /loss/FBt2/detection_loss‚2.5732918216122522 1loss/FBt4/assignment_loss‚0.24667755066429892 1loss/SRqqt1/detection_loss‚1.9213642539649174øž.’^¥Ø 2loss/SRqqt2/assignment_loss‚1.8533146729071934 /loss/FRt3/detection_loss‚1.6386001452400878 1loss/SRqqt3/detection_loss‚1.1423399933006453 _step‚114 /loss/FRt4/detection_loss‚2.9780624017995945 1loss/SRqqt4/detection_loss‚2.8344944071769715 &loss/total_loss‚51.153899523260996 .loss/FBt3/assignment_loss‚0.33556479958093 /loss/FBt2/detection_loss‚2.5732918216122522 1loss/FBt4/assignment_loss‚0.24667755066429892 1loss/SRqqt1/detection_loss‚1.9213642539649174 epoch‚1 0loss/FRt3/assignment_loss‚2.1400843979539097 .loss/FRt2/detection_loss‚2.945626337081194 0loss/SRqqt2/detection_loss‚3.444748419854376 /loss/FBt1/detection_loss‚1.0360009660306515 2loss/SRqqt3/assignment_loss‚1.1176971080348543 0loss/FBt1/assignment_loss‚0.2970232487378353 /loss/FBt3/detection_loss‚2.8831029401885138 % _timestamp‚1.7884975633380601e+09 0loss/FRt4/assignment_loss‚4.6935389041900635 .loss/FBt4/detection_loss‚2.995287024974823 trainer/global_step‚1399 _runtime‚1621.115051063 /loss/FRt2/assignment_loss‚4.400761805474758 2loss/SRqqt1/assignment_loss‚1.8320182644087692 2loss/SRqqt4/assignment_loss‚2.2975025218725205 0loss/FBt2/assignment_loss‚0.2722888867833212 /loss/FRt1/detection_loss‚2.1354327620578593 /loss/FRt1/assignment_loss‚3.139075889370658rø¥È¡’¦‚(:‡ ©éÔ»¬÷n gpu.0.gpu‚14.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.774' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.9435 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µGnñ`§‚(:Õ ©éÔª›ÿo disk.md0.in‚ 1.94140625 disk.rbd2.in‚0* proc.memory.percent‚18.263649940490723 disk.sda.out‚179.39013671875 disk.sda1.out‚0 disk.rbd0.out‚ 0.279296875 disk.md0p1.out‚ 172.51171875 disk.dm-0.out‚ 18802.2890625 disk.sdb.in‚ 0.26953125 disk.sdb2.in‚ 0.26953125! disk.sdb2.out‚179.39013671875! disk.sda2.out‚179.39013671875# memory_percent‚99.9640941619873# proc.memory.rssMB‚ 2992.31640625 cpu‚25.191005086462297 network.sent‚ 2.377816e+06% disk./.usageGB‚2995.3857803344727 disk.md0p1.in‚ 1.94140625% proc.memory.availableMB‚ 5.8828125 network.recv‚595460 disk.nvme0n1.in‚ 56.55859375 disk.nvme0n1.out‚ 7997.9375" disk.nvme1n1.out‚ 10804.3515625 disk.md0.out‚ 172.51171875 disk.nvme1n1.in‚ 9.0078125 disk.sda.in‚1.671875 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb.out‚179.39013671875 disk.sda2.in‚1.671875) disk./.usagePercent‚83.75224008449563 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.dm-0.in‚ 65.56640625 disk.rbd2.out‚ 0.56640625Ýeíñ“¨‚(:ˆ ¸éÔ××Èk gpu.0.gpu‚15.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚179.397' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.84925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z:þO©‚(:Ä ¸éÔøÇäm disk.rbd0.in‚0 disk.rbd0.out‚ 0.279296875 network.recv‚601494 disk.rbd2.in‚0) disk./.usagePercent‚83.75217555487369# disk.nvme1n1.out‚10859.30859375 disk.dm-0.in‚ 65.56640625 disk.dm-0.out‚19067.01953125 disk.sda2.in‚1.671875 disk.sda2.out‚181.4501953125 disk.nvme0n1.in‚ 56.55859375 disk.sdb2.in‚ 0.26953125 disk.sda1.in‚0 disk.sdb1.in‚0 disk.md0p1.in‚ 1.94140625! proc.memory.availableMB‚5.875 network.sent‚ 2.487349e+06 disk.nvme1n1.in‚ 9.0078125 disk.sdb2.out‚181.4501953125 disk.sdb.in‚ 0.26953125 cpu‚25.189513420224735$ disk./.usageGB‚2995.383472442627! disk.nvme0n1.out‚ 8207.7109375 disk.md0p1.out‚174.5$ memory_percent‚99.96414184570312* proc.memory.percent‚18.263649940490723 disk.sda.out‚181.4501953125 disk.sdb1.out‚0 disk.sdb.out‚181.4501953125# proc.memory.rssMB‚ 2992.31640625 proc.cpu.threads‚17 disk.md0.in‚ 1.94140625 disk.md0.out‚174.5 disk.rbd2.out‚ 0.56640625 disk.sda.in‚1.671875 disk.sda1.out‚0«òF“ª‚(:ˆ ÇéÔž»àk gpu.0.gpu‚32.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äÿÃb«‚(:× ÇéÔ·ÿm disk.rbd0.out‚ 0.279296875 disk.sdb.out‚182.92626953125 disk.nvme0n1.in‚ 56.55859375 disk.rbd2.in‚0% disk./.usageGB‚2995.3268127441406 disk.md0p1.out‚ 175.9140625 disk.dm-0.out‚ 19167.953125! disk.sda2.out‚182.92626953125# disk.nvme1n1.out‚10860.12109375 disk.md0.in‚ 1.94140625& proc.memory.availableMB‚ 5.62890625 cpu‚25.18814499353567! disk.sdb2.out‚182.92626953125 disk.rbd2.out‚ 0.56640625$ memory_percent‚99.96564388275146 proc.cpu.threads‚17 network.sent‚ 2.49095e+06 disk.sda.out‚182.92626953125 disk.sda1.in‚0) disk./.usagePercent‚83.75059132599009 disk.rbd0.in‚0 disk.md0.out‚ 175.9140625 disk.dm-0.in‚ 65.56640625 disk.sdb.in‚ 0.26953125" disk.nvme0n1.out‚ 8307.83203125* proc.memory.percent‚18.263649940490723 disk.md0p1.in‚ 1.94140625 disk.nvme1n1.in‚ 9.0078125 network.recv‚605755 disk.sda.in‚1.671875 disk.sda1.out‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda2.in‚1.671875# proc.memory.rssMB‚ 2992.31640625 disk.sdb2.in‚ 0.26953125û„”î2¬- _wandbruntime‚1677  _runtime‚1677)±®fÄ­¾ _runtime‚1677.044106103 _step‚115 % lr-AdamW/pg1‚0.001051090999320348 % lr-AdamW/pg2‚0.001051090999320348 trainer/global_step‚1449 $ _timestamp‚1.788497619266753e+09¡.7êȮ _runtime‚1677.044106103 _step‚115 % lr-AdamW/pg1‚0.001051090999320348 % lr-AdamW/pg2‚0.001051090999320348 trainer/global_step‚1449 $ _timestamp‚1.788497619266753e+09só ¤2¯- _wandbruntime‚1677  _runtime‚1677GÂÁ]À°åO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚123447924384768 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚123446996411392 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232977408 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66866653 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36648873 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1657046025728 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36648873 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1651755655005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30217780 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1735109825024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚117774191331216 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36675215 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚121789950385664 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚121789968388608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚615 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119425039011225 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1657046025728 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26936353 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _runtime‚ 1677.04575548 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256917504 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚134393141716480 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136128484518912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 Þ `Ã+cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1735342802432 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1657955996160 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚121789950385664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚136128227601408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚116 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66892997 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 % _timestamp‚1.7884976192679043e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚123446996411392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1657955996160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚134393117776384 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1650847680009 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚123447924384768 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚121789968388608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18103704 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚119425964025581Vœ)ï'±éO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36648873 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1657046025728 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36648873 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1651755655005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30217780 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1735109825024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚117774191331216 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36675215 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚121789950385664 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚121789968388608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚615 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119425039011225 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1657046025728 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26936353 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _runtime‚ 1677.04575548 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256917504 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚134393141716480 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136128484518912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1735342802432 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1657955996160 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚121789950385664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚136128227601408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚116 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66892997 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 % _timestamp‚1.7884976192679043e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚123446996411392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1657955996160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚134393117776384 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1650847680009 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚123447924384768 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚121789968388608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18103704 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚119425964025581 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚123447924384768 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚123446996411392 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232977408 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66866653tÀ¶Þ®2²- _wandbruntime‚1678  _runtime‚1678¼Î·ö&³ðM N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1659077469696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30237942 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26954127 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚117853225965120 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚121871587474432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8838358 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119505173879047 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66911644 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚134483367420928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36673704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚1678.036253262 _step‚117 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚123530664944128 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884976202584937e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30237940 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1658167485952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18115769 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252709888 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚601 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36700072 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228769792 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚119506098893611 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30237942 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36700072 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚599 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚121871569471488 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8838957 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚136219872133632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66938014 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8838358 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119505173879047 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66911644 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚134483367420928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36673704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚1678.036253262 _step‚117 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚123530664944128 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884976202584937e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30237940 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1658167485952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18115769 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252709888 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚601 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36700072 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228769792 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚119506098893611 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30237942 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36700072 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚599 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚121871569471488 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8838957 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚136219872133632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66938014 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 XIDSM¼_eviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1659077469696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30237942 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26954127 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0u;w´´2µ- _wandbruntime‚1678  _runtime‚1678ͦŠS¶Í /loss/FBt2/detection_loss‚3.0903988971429714 0loss/FBt3/assignment_loss‚0.6430416953646474 1loss/SRqqt1/assignment_loss‚1.726457413782676 .loss/FRt3/detection_loss‚1.784021924538859 1loss/SRqqt4/detection_loss‚2.8627412574631825 /loss/FRt1/assignment_loss‚2.836138805205172 1loss/SRqqt3/assignment_loss‚1.030821065021465 1loss/SRqqt3/detection_loss‚0.9929504059255123 .loss/FBt3/detection_loss‚3.208718419075012 %loss/total_loss‚50.85226024768969 1loss/SRqqt2/assignment_loss‚1.974650320730039 1loss/SRqqt1/detection_loss‚2.1190721760193507 /loss/FBt1/detection_loss‚0.8663243645519146 trainer/global_step‚1449 # _timestamp‚1.78849762026079e+09 0loss/FBt1/assignment_loss‚0.3014122246728496 0loss/FBt2/assignment_loss‚0.2377409972518913 0loss/FBt4/assignment_loss‚0.4682785069190787 /loss/FRt1/detection_loss‚2.2763811431782828 /loss/FRt2/assignment_loss‚4.714259274078138 0loss/FRt3/assignment_loss‚1.9539356889395878 /loss/FRt4/assignment_loss‚4.734237615551267 _runtime‚1678.037774729 1loss/SRqqt4/assignment_loss‚2.537824409348624 /loss/FRt4/detection_loss‚3.0616769109453474 1loss/SRqqt2/detection_loss‚2.4366621375083923 epoch‚1 /loss/FRt2/detection_loss‚3.0062255462010703 .loss/FBt4/detection_loss‚1.988289048274358 _step‚118štZW·Ñ 1loss/SRqqt4/detection_loss‚2.8627412574631825 /loss/FRt1/assignment_loss‚2.836138805205172 1loss/SRqqt3/assignment_loss‚1.030821065021465 1loss/SRqqt3/detection_loss‚0.9929504059255123 .loss/FBt3/detection_loss‚3.208718419075012 %loss/total_loss‚50.85226024768969 1loss/SRqqt2/assignment_loss‚1.974650320730039 1loss/SRqqt1/detection_loss‚2.1190721760193507 /loss/FBt1/detection_loss‚0.8663243645519146 trainer/global_step‚1449 # _timestamp‚1.78849762026079e+09 0loss/FBt1/assignment_loss‚0.3014122246728496 0loss/FBt2/assignment_loss‚0.2377409972518913 0loss/FBt4/assignment_loss‚0.4682785069190787 /loss/FRt1/detection_loss‚2.2763811431782828 /loss/FRt2/assignment_loss‚4.714259274078138 0loss/FRt3/assignment_loss‚1.9539356889395878 /loss/FRt4/assignment_loss‚4.734237615551267 _runtime‚1678.037774729 1loss/SRqqt4/assignment_loss‚2.537824409348624 /loss/FRt4/detection_loss‚3.0616769109453474 1loss/SRqqt2/detection_loss‚2.4366621375083923 epoch‚1 /loss/FRt2/detection_loss‚3.0062255462010703 .loss/FBt4/detection_loss‚1.988289048274358 _step‚118 /loss/FBt2/detection_loss‚3.0903988971429714 0loss/FBt3/assignment_loss‚0.6430416953646474 1loss/SRqqt1/assignment_loss‚1.726457413782676 .loss/FRt3/detection_loss‚1.784021924538859vÖ¿RÀ“¸‚(:ˆ ÖéÔ†—ám gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚179.397' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.84925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãìN^¹‚(:Ó ÖéÔâ«ÿm disk.sda2.out‚184.3388671875 disk.md0.in‚ 1.94140625* proc.memory.percent‚18.263649940490723 proc.cpu.threads‚17 network.sent‚ 2.510185e+06$ disk./.usageGB‚2995.411201477051 disk.rbd0.in‚0 disk.nvme0n1.in‚ 56.55859375 disk.sdb2.in‚ 0.26953125 disk.sdb.out‚184.3388671875 cpu‚25.186759573486526 disk.sdb1.out‚0 disk.sdb.in‚ 0.26953125 disk.rbd2.in‚0 disk.sdb2.out‚184.3388671875 disk.dm-0.out‚19268.73828125" disk.nvme1n1.out‚ 10903.4453125 disk.rbd2.out‚ 0.56640625# proc.memory.availableMB‚5.65625# proc.memory.rssMB‚ 2992.31640625 network.recv‚610696 disk.sda1.in‚0 disk.sdb1.in‚0 disk.dm-0.in‚ 65.56640625" disk.nvme0n1.out‚ 8365.29296875# memory_percent‚99.9654769897461 disk.md0.out‚ 177.26953125 disk.sda.in‚1.671875 disk.sda.out‚184.3388671875 disk.sda1.out‚0 disk.md0p1.in‚ 1.94140625 disk.sda2.in‚1.671875 disk.nvme1n1.in‚ 9.0078125) disk./.usagePercent‚83.75295087028168 disk.rbd0.out‚ 0.279296875 disk.md0p1.out‚ 177.26953125ñ‡}“º‚(:ˆ åéÔáÇÞk gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚170.891' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.72275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rà2—g»‚(:Ü åéÔÔÎÚl$ memory_percent‚99.96554851531982 cpu‚25.185315706436196 network.sent‚ 2.603956e+06 disk.md0.out‚ 179.24609375 disk.sda1.in‚0 disk.md0p1.out‚ 179.24609375 disk.dm-0.out‚19504.57421875 disk.nvme0n1.in‚ 56.55859375 disk.nvme1n1.in‚ 9.0078125# disk.nvme1n1.out‚10981.98828125 disk.md0p1.in‚ 1.94140625 disk.sdb.out‚186.36865234375) disk./.usagePercent‚83.75476580589626 disk.sdb1.out‚0 disk.sda2.in‚1.671875 disk.rbd2.in‚0& proc.memory.availableMB‚ 5.64453125% disk./.usageGB‚2995.4761123657227! disk.sdb2.out‚186.36865234375 disk.sda.in‚1.671875 disk.rbd0.in‚0 disk.rbd2.out‚ 0.56640625 proc.cpu.threads‚17 network.recv‚615616 disk.sda1.out‚0 disk.rbd0.out‚ 0.279296875# proc.memory.rssMB‚ 2992.31640625 disk.sda.out‚186.36865234375 disk.md0.in‚ 1.94140625* proc.memory.percent‚18.263649940490723! disk.nvme0n1.out‚ 8522.5859375 disk.sdb2.in‚ 0.26953125 disk.sdb1.in‚0 disk.dm-0.in‚ 65.56640625 disk.sdb.in‚ 0.26953125! disk.sda2.out‚186.36865234375Òe–‘¼‚(:† ôéÔô˵m gpu.0.gpu‚14.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚116.916' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.229 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i6ºd½‚(:Ù ôéÔ­¦än( disk./.usagePercent‚83.7600083843711% disk./.usageGB‚2995.6636123657227 disk.nvme0n1.in‚ 56.55859375# disk.nvme1n1.out‚11080.43359375# memory_percent‚99.9661922454834 cpu‚25.184083361322113 proc.cpu.threads‚17! disk.sda2.out‚187.81884765625 disk.sdb2.in‚ 0.26953125 disk.sda1.in‚0 disk.md0p1.in‚ 1.94140625% proc.memory.availableMB‚ 5.5390625* proc.memory.percent‚18.263649940490723 disk.md0.out‚ 180.63671875 disk.sdb1.in‚0 disk.md0p1.out‚ 180.63671875 disk.sdb.in‚ 0.26953125 disk.nvme1n1.in‚ 9.0078125 disk.dm-0.out‚ 19813.3984375 disk.sdb1.out‚0 disk.dm-0.in‚ 65.56640625# proc.memory.rssMB‚ 2992.31640625 network.sent‚ 2.612287e+06 disk.sdb.out‚187.81884765625 disk.sda2.in‚1.671875 disk.sda.in‚1.671875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 0.279296875" disk.nvme0n1.out‚ 8732.96484375 disk.md0.in‚ 1.94140625! disk.sdb2.out‚187.81884765625 network.recv‚620343 disk.rbd2.in‚0 disk.rbd2.out‚ 0.56640625 disk.sda.out‚187.81884765625”I*À“¾‚(:ˆ ƒžéÔÌëél gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚164.039' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.00975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖJx+]¿‚(:Ò ƒžéÔã´m network.sent‚ 2.615888e+06% disk./.usageGB‚2994.8605728149414 disk.sdb2.out‚ 189.291015625 disk.md0p1.in‚ 1.94140625 disk.nvme0n1.in‚ 56.55859375( disk./.usagePercent‚83.7375550624331# disk.nvme1n1.out‚11178.18359375 disk.sda1.in‚0" disk.nvme0n1.out‚ 8821.30859375 disk.dm-0.out‚ 19999.4921875$ memory_percent‚99.96626377105713 disk.sda2.out‚ 189.291015625 disk.nvme1n1.in‚ 9.0078125 disk.md0.in‚ 1.94140625 disk.sda1.out‚0 disk.sda2.in‚1.671875 disk.md0.out‚ 182.02734375 disk.sda.in‚1.671875 disk.rbd0.in‚0 disk.sdb1.out‚0& proc.memory.availableMB‚ 5.52734375 disk.rbd2.in‚0 disk.sdb2.in‚ 0.26953125 disk.sda.out‚ 189.291015625 disk.rbd0.out‚ 0.279296875 disk.sdb1.in‚0 disk.sdb.in‚ 0.26953125# proc.memory.rssMB‚ 2992.31640625* proc.memory.percent‚18.263649940490723 network.recv‚624273 disk.rbd2.out‚ 0.56640625 disk.md0p1.out‚ 182.02734375 disk.dm-0.in‚ 65.56640625 disk.sdb.out‚ 189.291015625 cpu‚25.18254675796943 proc.cpu.threads‚17NmÝ÷2À- _wandbruntime‚1731  _runtime‚1731¡S?HÇÁÁ % _timestamp‚1.7884976737485504e+09 _runtime‚1731.525766065 _step‚119 & lr-AdamW/pg1‚0.0010505103924471496 & lr-AdamW/pg2‚0.0010505103924471496 trainer/global_step‚1499†Ã›7ËÂÅ % _timestamp‚1.7884976737485504e+09 _runtime‚1731.525766065 _step‚119 & lr-AdamW/pg1‚0.0010505103924471496 & lr-AdamW/pg2‚0.0010505103924471496 trainer/global_step‚1499w:D½2Ã- _wandbruntime‚1731  _runtime‚1731$1Âóë'ÄåO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚138904402998784 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚602 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1793621456896 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37920117 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚127584038127616 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37893775 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9122008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚121725071058416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31225782 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚138904426938880 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 _runtime‚ 1731.52682203 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚125870904676864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18706954 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27828360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚125870922679808 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18706956 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222491648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27828964 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚140698024455680 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37893775 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚140698270887424 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31225782 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246431744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚604 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚127584038127616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚125870922679808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚123431795993533 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31225780 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37920117 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1714043421184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚127584966100992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31225780 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚123432721007889 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚1499 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚120 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1713133450752 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚127584966100992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1713133450752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚121725088097776 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1707632910113ó ëï'ÅéO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18706954 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27828360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚125870922679808 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18706956 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222491648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27828964 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚140698024455680 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37893775 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚140698270887424 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31225782 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246431744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚604 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚127584038127616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚125870922679808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚123431795993533 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31225780 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37920117 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1714043421184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚127584966100992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31225780 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚123432721007889 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚1499 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚120 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1713133450752 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚127584966100992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1713133450752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚121725088097776 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1707632910113 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚138904402998784 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚602 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1793621456896 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37920117 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚127584038127616 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37893775 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9122008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚121725071058416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31225782 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚138904426938880 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 _runtime‚ 1731.52682203 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚125870904676864xÖŸOT2Æ- _wandbruntime‚1732  _runtime‚1732ïükUgÇïM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69190916 I?DeviceStatsMonitor.on_train_batch_end/activeÒváÆŽ!.small_pool.current‚26368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69164546 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1714254910976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69190916 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1708750183599 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31245942 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27846163 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 trainer/global_step‚1499 _runtime‚1732.477301494 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚125952541765632 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140789413176320 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚138994628703232 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31245940 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69164546 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27846773 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9127144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚125952523762688 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788497674699412e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254807040 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37944974 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18719021 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69190916 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69164546 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1714254910976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69190916 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1708750183599 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31245942 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27846163 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 trainer/global_step‚1499 _runtime‚1732.477301494 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚125952541765632 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140789413176320 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚138994628703232 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31245940 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69164546 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27846773 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9127144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚125952523762688 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788497674699412e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254807040 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37944974 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18719021 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1763510165221 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39138677 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32233782 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1762602190225 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚131722007817216 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚1549 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1769220875776 _runtime‚1782.035807978 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39138677 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127438552975841 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚143415712161280 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚601 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129951858968064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1770130846208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228783104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131721079843840 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚131721079843840 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71398801 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9412329 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129951876971008 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚125675967824976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39165019 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28722535 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71372457 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚599 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚143415688221184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376Ú(ßað'×êO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1770130846208 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1852075797504 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚125675950785616 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129951858968064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1770130846208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228783104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131721079843840 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚131721079843840 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71398801 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9412329 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129951876971008 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚125675967824976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39165019 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28722535 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71372457 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚599 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚143415688221184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252723200 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1763510165221 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39138677 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32233782 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1762602190225 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚131722007817216 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚1549 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1769220875776 _runtime‚1782.035807978 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39138677 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127438552975841 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚143415712161280 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚601 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32253942 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚145359416245248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 trainer/global_step‚1549 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚125754985419520 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71443818 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1770342336000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39163508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9417995 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125754968380160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1764627438707 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32253942 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32253940 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9417390 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚143505913925632 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28740266 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚130033496056832 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131803820389888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1763719463503 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19322269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28739659 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚127518687843663 _step‚125 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚143505937865728 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 _runtime‚ 1783.09750964 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚130033478053888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19322271 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244321280 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127519612858227 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1853478379520 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚130033496056832 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71443818 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220381184 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39189876 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39163508 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 $ _timestamp‚1.788497725319581e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚130033478053888 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚145359171923968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1853257998336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28739659 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚127518687843663 _step‚125 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚143505937865728 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 _runtime‚ 1783.09750964 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚130033478053888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19322271 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244321280 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127519612858227 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1853478379520 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚130033496056832 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71443818 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220381184 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39189876 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39163508 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 $ _timestamp‚1.788497725319581e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚130033478053888 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚145359171923968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1853257998336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32253942 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚145359416245248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 trainer/global_step‚1549 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚125754985419520 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71443818 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1770342336000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39163508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9417995 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125754968380160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1764627438707 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32253942 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32253940 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool(¡ÁÐ.peak‚32004 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9417390 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚143505913925632 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28740266 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚130033496056832 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131803820389888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1763719463503 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0}§îÄ2Û- _wandbruntime‚1783  _runtime‚1783LfZÜÔ _runtime‚1783.098653835 _step‚126 1loss/SRqqt4/detection_loss‚2.5039641495906944 .loss/FRt2/assignment_loss‚4.69827713171641 0loss/FRt3/assignment_loss‚1.7119878495948901 /loss/FRt1/detection_loss‚2.0351476535061375 trainer/global_step‚1549 /loss/FRt4/assignment_loss‚4.464928448200226 1loss/SRqqt1/detection_loss‚2.0284940726481953 1loss/FBt1/assignment_loss‚0.27102250432940667 .loss/FRt4/detection_loss‚3.303741921981176 1loss/SRqqt3/detection_loss‚1.0663081993850378 /loss/FBt3/detection_loss‚2.9958792209625242 2loss/SRqqt2/assignment_loss‚2.3150548304502783 /loss/FBt1/detection_loss‚0.9803179996772943 % _timestamp‚1.7884977253216374e+09 0loss/FRt1/assignment_loss‚3.2477417743206023 2loss/SRqqt1/assignment_loss‚1.9641256091686397 0loss/FBt2/assignment_loss‚0.3312700898386236 epoch‚1 1loss/SRqqt4/assignment_loss‚2.202374127552365 0loss/FBt3/assignment_loss‚0.3060679570189677 1loss/FBt4/assignment_loss‚0.25819184841501347 .loss/FRt2/detection_loss‚2.934840283791224 /loss/FRt3/detection_loss‚1.5904785312604213 1loss/SRqqt2/detection_loss‚2.6335606299913845 .loss/FBt2/detection_loss‚2.657552033662796 2loss/SRqqt3/assignment_loss‚1.1605233331748213 /loss/FBt4/detection_loss‚2.9556523944650377 %loss/total_loss‚50.617502594702171‹~O^ÝØ .loss/FRt2/assignment_loss‚4.69827713171641 0loss/FRt3/assignment_loss‚1.7119878495948901 /loss/FRt1/detection_loss‚2.0351476535061375 trainer/global_step‚1549 /loss/FRt4/assignment_loss‚4.464928448200226 1loss/SRqqt1/detection_loss‚2.0284940726481953 1loss/FBt1/assignment_loss‚0.27102250432940667 .loss/FRt4/detection_loss‚3.303741921981176 1loss/SRqqt3/detection_loss‚1.0663081993850378 /loss/FBt3/detection_loss‚2.9958792209625242 2loss/SRqqt2/assignment_loss‚2.3150548304502783 /loss/FBt1/detection_loss‚0.9803179996772943 % _timestamp‚1.7884977253216374e+09 0loss/FRt1/assignment_loss‚3.2477417743206023 2loss/SRqqt1/assignment_loss‚1.9641256091686397 0loss/FBt2/assignment_loss‚0.3312700898386236 epoch‚1 1loss/SRqqt4/assignment_loss‚2.202374127552365 0loss/FBt3/assignment_loss‚0.3060679570189677 1loss/FBt4/assignment_loss‚0.25819184841501347 .loss/FRt2/detection_loss‚2.934840283791224 /loss/FRt3/detection_loss‚1.5904785312604213 1loss/SRqqt2/detection_loss‚2.6335606299913845 .loss/FBt2/detection_loss‚2.657552033662796 2loss/SRqqt3/assignment_loss‚1.1605233331748213 /loss/FBt4/detection_loss‚2.9556523944650377 %loss/total_loss‚50.61750259470217 _runtime‚1783.098653835 _step‚126 1loss/SRqqt4/detection_loss‚2.5039641495906944~§×íö“Þ‚(:ˆ ¿žéÔ±Ùél gpu.0.gpu‚46.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.051' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.01275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0siv¡\ß‚(:Ñ ¿žéÔî÷Ìm disk.sdb2.in‚ 0.26953125 disk.sda1.out‚0& proc.memory.availableMB‚ 5.05859375 network.recv‚642788 disk.sdb.in‚ 0.26953125$ memory_percent‚99.96912479400635# disk.nvme1n1.out‚11560.24609375 disk.rbd2.out‚ 0.56640625! disk.sdb2.out‚196.14794921875 disk.sda.in‚1.671875 disk.sda1.in‚0 disk.dm-0.in‚ 65.56640625 disk.sdb.out‚196.14794921875$ disk./.usageGB‚2994.380516052246 disk.rbd0.out‚ 0.279296875* proc.memory.percent‚18.263649940490723) disk./.usagePercent‚83.72413247442879 disk.md0.out‚188.625 disk.nvme0n1.in‚ 56.55859375 disk.md0.in‚ 1.94140625 disk.sda.out‚196.14794921875 disk.sdb1.out‚0 disk.md0p1.in‚ 1.94140625 disk.sda2.in‚1.671875 cpu‚25.176905424755645 proc.cpu.threads‚17 network.sent‚ 2.740839e+06 disk.nvme1n1.in‚ 9.0078125 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.md0p1.out‚188.625! disk.sda2.out‚196.14794921875" disk.nvme0n1.out‚ 8894.04296875 disk.dm-0.out‚ 20454.2890625# proc.memory.rssMB‚ 2992.31640625 disk.rbd2.in‚0ÈQ‹’à‚(:‡ ΞéÔþØ„m gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.118' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.0295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jl laá‚(:Ö ΞéÔÊš¹m disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.in‚ 1.94140625# proc.memory.rssMB‚ 2992.31640625* proc.memory.percent‚18.263649940490723% disk./.usageGB‚2994.3278465270996 disk.sdb2.in‚ 0.26953125 disk.sda.out‚197.98388671875 disk.nvme0n1.in‚ 56.55859375 cpu‚25.175866840023215 network.recv‚649234 disk.sdb1.in‚0! disk.sda2.out‚197.98388671875) disk./.usagePercent‚83.72265981246177 disk.sda.in‚1.671875 disk.rbd2.out‚ 0.56640625 proc.cpu.threads‚17 network.sent‚ 2.848583e+06 disk.sdb.in‚ 0.26953125 disk.sda2.in‚1.671875 disk.md0.in‚ 1.94140625 disk.sdb1.out‚0 disk.dm-0.out‚20527.42578125 disk.sdb.out‚197.98388671875# disk.nvme1n1.out‚11561.20703125 disk.rbd2.in‚0& proc.memory.availableMB‚ 5.25390625! disk.sdb2.out‚197.98388671875 disk.rbd0.in‚0 disk.dm-0.in‚ 65.56640625 disk.nvme0n1.out‚ 8966.21875 disk.md0.out‚ 190.390625$ memory_percent‚99.96793270111084 disk.rbd0.out‚ 0.279296875 disk.md0p1.out‚ 190.390625 disk.nvme1n1.in‚ 9.0078125ò˜’„Žâ‚(:ƒ ÝžéÔ¿Ãõl gpu.0.gpu‚8.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚157.52' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.38 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}îåXã‚(:Í ÝžéÔ½˜n$ disk./.usageGB‚2994.477382659912 disk.dm-0.in‚ 65.56640625 disk.sdb.out‚ 199.52734375# disk.nvme1n1.out‚11561.31640625 disk.sdb1.in‚0 network.recv‚652918 disk.md0p1.out‚ 191.86328125 disk.dm-0.out‚ 20618.2109375 disk.sda1.in‚0 disk.nvme1n1.in‚ 9.0078125 disk.rbd2.in‚0 disk.sdb2.out‚ 199.52734375" disk.nvme0n1.out‚ 9056.89453125 disk.sda.in‚1.671875 proc.cpu.threads‚17 network.sent‚ 2.853822e+06 disk.md0p1.in‚ 1.94140625 disk.sda2.out‚ 199.52734375 disk.sda.out‚ 199.52734375 disk.sdb1.out‚0 disk.md0.out‚ 191.86328125 disk.rbd0.in‚0 disk.rbd0.out‚ 0.279296875% proc.memory.availableMB‚ 5.2421875" proc.memory.rssMB‚ 2992.3203125) disk./.usagePercent‚83.72684090532094 disk.sdb.in‚ 0.26953125 disk.sda2.in‚1.671875 disk.md0.in‚ 1.94140625 disk.rbd2.out‚ 0.56640625 disk.sdb2.in‚ 0.26953125 disk.nvme0n1.in‚ 56.55859375 disk.sda1.out‚0$ memory_percent‚99.96800422668457* proc.memory.percent‚18.263673782348633 cpu‚25.174684108625325L¬iä‚(:’ ìžéÔ¯ïm gpu.0.gpu‚48.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚119.368' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.841999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00x‹2Oå‚(:Ä ìžéÔ¿ó§m disk.rbd2.in‚0 disk.rbd2.out‚ 0.56640625 disk.sda1.out‚0 disk.rbd0.in‚0 disk.md0p1.in‚ 1.94140625 network.sent‚ 2.859812e+06 disk.sda1.in‚0 disk.sdb.in‚ 0.26953125 disk.sda2.out‚201.1748046875 disk.nvme1n1.out‚ 11561.84375 disk.md0.in‚ 1.94140625 disk.sdb2.in‚ 0.26953125 disk.sda.out‚201.1748046875 disk.dm-0.in‚ 65.56640625 disk.dm-0.out‚20802.82421875" disk.nvme0n1.out‚ 9240.98046875 disk.md0.out‚193.4375* proc.memory.percent‚18.263673782348633 cpu‚25.173431254927866 disk.rbd0.out‚ 0.279296875 disk.sda2.in‚1.671875 disk.nvme1n1.in‚ 9.0078125" proc.memory.rssMB‚ 2992.3203125 network.recv‚657460 disk.sdb1.in‚0 disk.md0p1.out‚193.4375# memory_percent‚99.9664306640625 proc.cpu.threads‚17 disk.sdb2.out‚201.1748046875 disk.sda.in‚1.671875 disk.sdb1.out‚0 disk.sdb.out‚201.1748046875) disk./.usagePercent‚83.73557608292012$ disk./.usageGB‚2994.789794921875 disk.nvme0n1.in‚ 56.55859375 proc.memory.availableMB‚5.5PQ 2æ- _wandbruntime‚1836  _runtime‚1836&Ð{Åç¿ trainer/global_step‚1599 $ _timestamp‚1.788497778896815e+09 _runtime‚ 1836.67406032 _step‚127 & lr-AdamW/pg1‚0.0010490215115081167 & lr-AdamW/pg2‚0.00104902151150811679D6^Éèà & lr-AdamW/pg2‚0.0010490215115081167 trainer/global_step‚1599 $ _timestamp‚1.788497778896815e+09 _runtime‚ 1836.67406032 _step‚127 & lr-AdamW/pg1‚0.0010490215115081167µýŠ›2é- _wandbruntime‚1836  _runtime‚1836•½ûì'êæO P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1826218271232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1910529162752 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚614 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19913456 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚134032831262208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚135859049533440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33241782 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33241780 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1825308300800 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚134032813259264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40383579 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238043136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40409921 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29614267 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚147926973443584 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1826218271232 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚149837502606336 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135858121560064 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚135859049533440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33241782 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33241780 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1825308300800 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚134032813259264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40383579 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238043136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40409921 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29614267 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚147926973443584 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1826218271232 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚149837502606336 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135858121560064 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73696720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19925521 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33261940 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9707272 _KDeviceStatsMonitor.on_train_batch_en4š)üñ!d/inactive_split_bytes.large_pool.freed‚148017199148032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73696720 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1911700622848 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1819596718611 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚134114432345088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚129705865146720 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚134114450348032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚131526369840535 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224575488 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚134114450348032 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚129705848107360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚135941790092800 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚148017223088128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33261940 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚135940862106112 trainer/global_step‚1599 _runtime‚1837.677794987 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33261942 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1826429761024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1827339744768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33261942 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29632171 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40434778 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33261942 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1826429761024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1827339744768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33261942 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29632171 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40434778 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73696720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19925521 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33261940 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9707272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚148017199148032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73696720 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1911700622848 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1819596718611 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚134114432345088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚129705865146720 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚134114450348032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚131526369840535 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224575488 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚134114450348032 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚129705848107360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚135941790092800 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚148017223088128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33261940 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚135940862106112 trainer/global_step‚1599 _runtime‚1837.677794987Ø2ï- _wandbruntime‚1837  _runtime‚1837>òQðË 2loss/SRqqt3/assignment_loss‚1.1103519616507977 0loss/FBt1/assignment_loss‚0.2249432935098438 .loss/FRt3/detection_loss‚1.750982231474009 /loss/FBt1/detection_loss‚0.6842624089147158 -loss/FBt2/detection_loss‚2.80383043885231 _runtime‚1837.678953432 /loss/FRt3/assignment_loss‚1.997372701548148 0loss/FBt4/assignment_loss‚0.3657848051516339 /loss/FRt1/detection_loss‚1.7219641447205059 0loss/SRqqt4/detection_loss‚2.346081871438671 /loss/FRt1/assignment_loss‚2.649813338201873 /loss/FRt2/assignment_loss‚3.922954633500841 0loss/FBt2/assignment_loss‚0.2508079988880979 0loss/FBt3/assignment_loss‚0.5078414054379599 1loss/SRqqt1/detection_loss‚2.4122307532363467 /loss/FRt2/detection_loss‚2.3905180546972487 %loss/total_loss‚48.84745752257304 2loss/SRqqt2/assignment_loss‚1.9537020888593462 1loss/SRqqt4/assignment_loss‚2.088100809503246 /loss/FRt4/detection_loss‚3.0283347943733476 2loss/SRqqt1/assignment_loss‚1.3285123385764934 /loss/SRqqt3/detection_loss‚1.27164579123118 epoch‚1 trainer/global_step‚1599 _step‚130 /loss/FRt4/assignment_loss‚4.412397956026012 $ _timestamp‚1.788497779901763e+09 0loss/SRqqt2/detection_loss‚3.326831022898356 .loss/FBt3/detection_loss‚3.268447296960013 /loss/FBt4/detection_loss‚3.0297453829220364ÖÝ‘VñÐ 2loss/SRqqt2/assignment_loss‚1.9537020888593462 1loss/SRqqt4/assignment_loss‚2.088100809503246 /loss/FRt4/detection_loss‚3.0283347943733476 2loss/SRqqt1/assignment_loss‚1.3285123385764934 /loss/SRqqt3/detection_loss‚1.27164579123118 epoch‚1 trainer/global_step‚1599 _step‚130 /loss/FRt4/assignment_loss‚4.412397956026012 $ _timestamp‚1.788497779901763e+09 0loss/SRqqt2/detection_loss‚3.326831022898356 .loss/FBt3/detection_loss‚3.268447296960013 /loss/FBt4/detection_loss‚3.0297453829220364 2loss/SRqqt3/assignment_loss‚1.1103519616507977 0loss/FBt1/assignment_loss‚0.2249432935098438 .loss/FRt3/detection_loss‚1.750982231474009 /loss/FBt1/detection_loss‚0.6842624089147158 -loss/FBt2/detection_loss‚2.80383043885231 _runtime‚1837.678953432 /loss/FRt3/assignment_loss‚1.997372701548148 0loss/FBt4/assignment_loss‚0.3657848051516339 /loss/FRt1/detection_loss‚1.7219641447205059 0loss/SRqqt4/detection_loss‚2.346081871438671 /loss/FRt1/assignment_loss‚2.649813338201873 /loss/FRt2/assignment_loss‚3.922954633500841 0loss/FBt2/assignment_loss‚0.2508079988880979 0loss/FBt3/assignment_loss‚0.5078414054379599 1loss/SRqqt1/detection_loss‚2.4122307532363467 /loss/FRt2/detection_loss‚2.3905180546972487 %loss/total_loss‚48.84745752257304‚ûøQ“ò‚(:ˆ ûžéÔ¤þín gpu.0.gpu‚34.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.545' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.13625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~ñÀŠ_ó‚(:Ô ûžéÔçúo disk.sda.in‚1.671875 disk.md0p1.in‚ 1.94140625* proc.memory.percent‚18.263673782348633 disk.sda2.in‚1.671875& proc.memory.availableMB‚ 5.59765625) disk./.usagePercent‚83.74605430693522 disk.md0.out‚ 194.8046875 disk.sda1.out‚0 disk.sdb1.out‚0 disk.nvme0n1.in‚ 56.55859375 disk.sdb2.in‚ 0.26953125 disk.rbd0.out‚ 0.279296875 disk.md0p1.out‚ 194.8046875 disk.dm-0.in‚ 65.56640625 disk.nvme1n1.in‚ 9.0078125" disk.nvme1n1.out‚ 11561.9609375 disk.md0.in‚ 1.94140625 disk.rbd2.out‚ 0.56640625 disk.sdb1.in‚0" disk.nvme0n1.out‚ 9495.91796875% disk./.usageGB‚2995.1645469665527 disk.sdb2.out‚202.6103515625 disk.sda.out‚202.6103515625 disk.rbd0.in‚0 disk.sdb.in‚ 0.26953125 disk.sdb.out‚202.6103515625" proc.memory.rssMB‚ 2992.3203125 network.recv‚661575 disk.rbd2.in‚0 disk.sda2.out‚202.6103515625$ memory_percent‚99.96583461761475 cpu‚25.17223339690928 proc.cpu.threads‚17 network.sent‚ 2.865734e+06 disk.sda1.in‚0 disk.dm-0.out‚21057.87890625SÂõX“ô‚(:ˆ ŠŸéÔ—»íl gpu.0.gpu‚12.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚181.699' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.42475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Öécžeõ‚(:Ú ŠŸéÔ×Í¿m* proc.memory.percent‚18.263673782348633 proc.cpu.threads‚17 disk.rbd0.out‚ 0.279296875 disk.dm-0.out‚21237.00390625" disk.nvme0n1.out‚ 9674.82421875 disk.md0.in‚ 1.94140625 disk.sdb2.in‚ 0.26953125" proc.memory.rssMB‚ 2992.3203125 disk.rbd0.in‚0 disk.dm-0.in‚ 65.56640625" disk.nvme1n1.out‚ 11562.1796875% disk./.usageGB‚2995.4155464172363 disk.sda1.out‚0 disk.sdb.in‚ 0.26953125! disk.sdb2.out‚204.52099609375% proc.memory.availableMB‚ 5.0546875 disk.md0p1.out‚ 196.64453125 disk.rbd2.out‚ 0.56640625 disk.md0p1.in‚ 1.94140625 disk.rbd2.in‚0 network.recv‚667812 disk.sda1.in‚0 disk.sdb1.in‚0 disk.sda2.in‚1.671875 disk.nvme1n1.in‚ 9.0078125 disk.sda.in‚1.671875 disk.sda.out‚204.52099609375 network.sent‚ 2.975551e+06 disk.sdb1.out‚0 disk.sdb.out‚204.52099609375 disk.nvme0n1.in‚ 56.55859375$ memory_percent‚99.96914863586426 cpu‚25.171064125508934) disk./.usagePercent‚83.75307235662777! disk.sda2.out‚204.52099609375 disk.md0.out‚ 196.64453125ûì r“ö‚(:ˆ ™ŸéÔ¯÷ül gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øšÅ ^÷‚(:Ó ™ŸéÔ䔤n" proc.memory.rssMB‚ 2992.3203125 disk.md0.in‚ 1.94140625 network.recv‚671513! disk.sdb2.out‚205.91943359375 disk.sda.in‚1.671875 disk.sda1.out‚0 disk.dm-0.in‚ 65.56640625 disk.dm-0.out‚21463.34765625 disk.rbd2.in‚0 disk.sdb.out‚205.91943359375$ memory_percent‚99.96380805969238* proc.memory.percent‚18.263673782348633 disk.rbd0.out‚ 0.279296875% disk./.usageGB‚2995.9145278930664 disk.sdb2.in‚ 0.26953125 disk.sda2.in‚1.671875 disk.nvme1n1.in‚ 9.0078125" disk.nvme1n1.out‚ 11562.2890625 disk.rbd2.out‚ 0.56640625 disk.sdb.in‚ 0.26953125! disk.sda2.out‚205.91943359375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 56.55859375" disk.nvme0n1.out‚ 9901.05859375 cpu‚25.16982389295424 network.sent‚ 2.978715e+06 disk.sda1.in‚0 disk.md0p1.out‚ 197.96875 disk.md0.out‚ 197.96875% proc.memory.availableMB‚ 5.9296875 proc.cpu.threads‚17) disk./.usagePercent‚83.76702408753195 disk.sda.out‚205.91943359375 disk.md0p1.in‚ 1.94140625l/—“ø‚(:ˆ ¨ŸéÔăÄm gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.481' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.62025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>*¥Hù‚(:½ ¨ŸéÔ©¸¨n disk.sda1.out‚0 network.recv‚676236 disk.md0.out‚ 199.42578125 disk.rbd0.out‚ 0.279296875) disk./.usagePercent‚83.76702408753195 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0p1.out‚ 199.42578125 disk.sdb2.out‚207.4375 cpu‚25.168753856552083 proc.cpu.threads‚17% disk./.usageGB‚2995.9145278930664 disk.sdb.in‚ 0.26953125 disk.rbd2.out‚ 0.56640625 disk.sdb2.in‚ 0.26953125$ memory_percent‚99.96683597564697" proc.memory.rssMB‚ 2992.3203125* proc.memory.percent‚18.263673782348633 network.sent‚ 2.984705e+06 disk.sda2.in‚1.671875 disk.nvme0n1.in‚ 56.55859375! disk.nvme0n1.out‚ 10168.734375 disk.nvme1n1.in‚ 9.0078125 disk.md0.in‚ 1.94140625 disk.sda.in‚1.671875 disk.rbd0.in‚0 disk.md0p1.in‚ 1.94140625 disk.dm-0.in‚ 65.56640625 disk.dm-0.out‚ 21731.1796875 disk.sdb.out‚207.4375 disk.sda.out‚207.4375& proc.memory.availableMB‚ 5.43359375 disk.sda2.out‚207.4375" disk.nvme1n1.out‚ 11562.4453125 disk.rbd2.in‚0 disk.sda1.in‚0 …U 2ú- _wandbruntime‚1891  _runtime‚1891;wm4Åû¿ % lr-AdamW/pg1‚0.001048113547722838 % lr-AdamW/pg2‚0.001048113547722838 trainer/global_step‚1649 % _timestamp‚1.7884978336387591e+09 _runtime‚1891.416021878 _step‚131Ç‘ß|ÊüÄ % lr-AdamW/pg1‚0.001048113547722838 % lr-AdamW/pg2‚0.001048113547722838 trainer/global_step‚1649 % _timestamp‚1.7884978336387591e+09 _runtime‚1891.416021878 _step‚131ƒöD?2ý- _wandbruntime‚1891  _runtime‚1891ñµô«þåO SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9992118 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139995163276288 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226685952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20516704 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34249780 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1881395725824 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚154407459795456 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41628481 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34249782 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚138113785553408 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚138113767550464 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocatedµ…«âׂ 1882305696256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30508822 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚135452991954813 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚133577727279376 _step‚132 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1969177189376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1882305696256 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75904605 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34249780 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41654823 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41628481 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20516706 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚135452066940457 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚133577710240016 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚138113767550464 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1881395725824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9992741 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚154407209169408 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _runtime‚1891.417266914 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34249782 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250626048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30509447 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41654823 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚138113785553408òiªð'ÿêO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41628481 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34249782 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚138113785553408 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚138113767550464 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1882305696256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30508822 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚135452991954813 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚133577727279376 _step‚132 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1969177189376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1882305696256 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75904605 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34249780 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41654823 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41628481 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20516706 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚135452066940457 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚133577710240016 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚138113767550464 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1881395725824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9992741 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚154407209169408 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _runtime‚1891.417266914 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34249782 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250626048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30509447 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41654823 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚138113785553408 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9992118 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139995163276288 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226685952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20516704 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34249780 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1881395725824 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚154407459795456 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185„)Y¿2€- _wandbruntime‚1892  _runtime‚1892D©`ö&ðM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1883427169792 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20528771 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 % _timestamp‚1.7884978346295671e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75949622 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41653312 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34269940 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1883427169792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1970118999040 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚135533126822843 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30526651 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1882517186048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚152528484370432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 trainer/global_step‚1649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚138195386636288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚140078831809024 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9997882 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚133656727834560 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725…N!3î‚õM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34269942 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚154498603369472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1883427169792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1970118999040 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚135533126822843 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30526651 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1882517186048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚152528484370432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 trainer/global_step‚1649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚138195386636288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚140078831809024 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9997882 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚133656727834560 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1883427169792 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20528771 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 % _timestamp‚1.7884978346295671e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75949622 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41653312 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34269940 F¢ÌŽÆ & lr-AdamW/pg2‚0.0010470968131157375 trainer/global_step‚1699 % _timestamp‚1.7884978882274334e+09 _runtime‚1946.004708911 _step‚135 & lr-AdamW/pg1‚0.0010470968131157375‡e$'æ2- _wandbruntime‚1946  _runtime‚1946šƒ’ î'èO A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚142194721841664 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144133132965888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10283911 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2027704499712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42899725 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31404482 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144133132965888 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144132204992512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1938393121280 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42873383 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42873383 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚142194739844608 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚137528589967216 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1930233955549 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10284526 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚78157507 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31403865 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚139459748937121 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78131163 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _step‚136 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚142194721841664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1937483150848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884978882285576e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1937483150848 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35257780 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256917504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚1946.006024817 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 trainer/global_step‚1699 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35257782 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760¤HOoó'‘íO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42899725 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31404482 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144133132965888 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144132204992512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1938393121280 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42873383 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42873383 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚142194739844608 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚137528589967216 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1930233955549 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10284526 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚78157507 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31403865 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚139459748937121 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78131163 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _step‚136 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚142194721841664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1937483150848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884978882285576e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1937483150848 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35257780 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256917504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚1946.006024817 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 trainer/global_step‚1699 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35257782 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚142194721841664 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144133132965888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10283911 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2027704499712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144ˆJ´2’- _wandbruntime‚1947  _runtime‚1947g£@¥“ðM VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1699 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21132019 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78202524 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚159068405299712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚142276340927488 M=DeviceStatsMonitor.on_train_batch_end/rãd=¶Q"eserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚142276358930432 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35277940 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21132021 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78176154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31422196 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚144215873525248 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚139538958790587 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31421576 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42898214 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚142276358930432 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1939514594816 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42924582 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252709888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚157039769592832 _runtime‚ 1947.06450365 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1938604611072 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2028864476672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78202524 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚137607607561760 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 _step‚137 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚144215873525248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚142276340927488 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35277940 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚159068658009600 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚137607624601120 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚144214945538560 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2028635706880 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1932259204031 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78176154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0šp§šû&”õM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78202524 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚137607607561760 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 _step‚137 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚144215873525248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚142276340927488 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35277940 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚159068658009600 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚137607624601120 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚144214945538560 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2028635706880 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1932259204031 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78176154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1699 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21132019 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78202524 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚159068405299712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚142276340927488 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚142276358930432 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35277940 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21132021 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78176154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31422196 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚144215873525248 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚139538958790587 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31421576 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42898214 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚142276358930432 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1939514594816 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42924582 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252709888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚157039769592832 _runtime‚ 1947.06450365 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1938604611072 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2028864476672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144‰±ÆÞ2•- _wandbruntime‚1947  _runtime‚1947\rGnY–Ó .loss/FBt3/detection_loss‚3.095742776989937 0loss/FRt1/assignment_loss‚2.4767239892736392 0loss/SRqqt1/detection_loss‚2.417512454665624 1loss/SRqqt4/detection_loss‚2.5705027268046425 /loss/FBt1/detection_loss‚1.1382735572479905 epoch‚1 % _timestamp‚1.7884978892884762e+09 /loss/FRt2/detection_loss‚2.4100884415391013 /loss/FBt4/detection_loss‚2.3352819005648295 _step‚138 /loss/FRt2/assignment_loss‚4.266314204146222 1loss/SRqqt1/assignment_loss‚2.148704670369625 1loss/FBt3/assignment_loss‚0.06410225237414124 1loss/FBt4/assignment_loss‚0.32626792677295574 2loss/SRqqt2/assignment_loss‚1.9888031773078136 0loss/FBt2/assignment_loss‚0.4395022951212013 /loss/FRt4/detection_loss‚3.6069996742641224 0loss/FRt3/assignment_loss‚1.8210027891656626 /loss/SRqqt3/detection_loss‚0.97272986326028 _runtime‚1947.065475414 1loss/SRqqt4/assignment_loss‚2.056155104012716 /loss/FRt3/detection_loss‚1.7307751424934552 /loss/FRt4/assignment_loss‚3.896991154726814 /loss/FRt1/detection_loss‚1.8551843007388673 1loss/SRqqt2/detection_loss‚2.3368074802251964 .loss/FBt2/detection_loss‚2.661054529249668 %loss/total_loss‚48.03178207494973 trainer/global_step‚1699 2loss/SRqqt3/assignment_loss‚1.1889333869753913 1loss/FBt1/assignment_loss‚0.22732827665983424G„=^—Ø /loss/SRqqt3/detection_loss‚0.97272986326028 _runtime‚1947.065475414 1loss/SRqqt4/assignment_loss‚2.056155104012716 /loss/FRt3/detection_loss‚1.7307751424934552 /loss/FRt4/assignment_loss‚3.896991154726814 /loss/FRt1/detection_loss‚1.8551843007388673 1loss/SRqqt2/detection_loss‚2.3368074802251964 .loss/FBt2/detection_loss‚2.661054529249668 %loss/total_loss‚48.03178207494973 trainer/global_step‚1699 2loss/SRqqt3/assignment_loss‚1.1889333869753913 1loss/FBt1/assignment_loss‚0.22732827665983424 .loss/FBt3/detection_loss‚3.095742776989937 0loss/FRt1/assignment_loss‚2.4767239892736392 0loss/SRqqt1/detection_loss‚2.417512454665624 1loss/SRqqt4/detection_loss‚2.5705027268046425 /loss/FBt1/detection_loss‚1.1382735572479905 epoch‚1 % _timestamp‚1.7884978892884762e+09 /loss/FRt2/detection_loss‚2.4100884415391013 /loss/FBt4/detection_loss‚2.3352819005648295 _step‚138 /loss/FRt2/assignment_loss‚4.266314204146222 1loss/SRqqt1/assignment_loss‚2.148704670369625 1loss/FBt3/assignment_loss‚0.06410225237414124 1loss/FBt4/assignment_loss‚0.32626792677295574 2loss/SRqqt2/assignment_loss‚1.9888031773078136 0loss/FBt2/assignment_loss‚0.4395022951212013 /loss/FRt4/detection_loss‚3.6069996742641224 0loss/FRt3/assignment_loss‚1.8210027891656626ŠO7a哘‚(:ˆ äŸéÔï˜ðq gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚142.313' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.57825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªî‹ve™‚(:Ú äŸéÔ¯Üét disk.rbd2.in‚0 disk.sdb1.in‚0# disk.nvme1n1.out‚11652.26953125 cpu‚25.164342672139167 disk.md0p1.out‚ 205.37890625 disk.dm-0.out‚22462.43359375 disk.rbd2.out‚ 0.56640625* proc.memory.percent‚18.263697624206543 disk.rbd0.in‚0 network.recv‚693853# disk./.usageGB‚2995.41459274292 disk.sda.out‚213.63818359375! disk.sda2.out‚213.63818359375 disk.dm-0.in‚ 65.56640625 disk.nvme0n1.in‚ 56.55859375 disk.nvme1n1.in‚ 9.0078125& proc.memory.availableMB‚ 5.80859375 disk.rbd0.out‚ 0.279296875 disk.sdb1.out‚0 network.sent‚ 3.108723e+06) disk./.usagePercent‚83.75304569149473! disk.sdb2.out‚213.63818359375 disk.sda1.in‚0 disk.md0p1.in‚ 1.94140625 disk.sdb.out‚213.63818359375 disk.sdb2.in‚ 0.26953125 disk.sda.in‚1.671875" disk.nvme0n1.out‚ 10810.1640625 disk.md0.in‚ 1.94140625# memory_percent‚99.9645471572876# proc.memory.rssMB‚ 2992.32421875 proc.cpu.threads‚17 disk.md0.out‚ 205.37890625 disk.sda1.out‚0 disk.sdb.in‚ 0.26953125 disk.sda2.in‚1.671875-ÿrQ’š‚(:‡ óŸéÔýÌðl gpu.0.gpu‚24.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚183.258' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.8145 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚146275694135808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44144627 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚143466505919429 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚148270174682112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21723206 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚606 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚161460853050880 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2086364457984 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocatedGÇÃ@‚44144627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚141479469694416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44118285 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚146275676132864 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚163546979465728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚161460829110784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1987019185653 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1994480546304 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36265780 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10574723 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238043136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214103040 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1993570575872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80384065 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80410409 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚143465580905073 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _step‚140 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36265780 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36265782 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚146275694135808 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148269246708736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚148270174682112 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚1749 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21723204 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168ªP©Kó'¥íO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚141479486733776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1993570575872 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚146275694135808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44144627 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚143466505919429 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚148270174682112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21723206 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚606 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚161460853050880 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2086364457984 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44144627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚141479469694416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44118285 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚146275676132864 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚163546979465728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚161460829110784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1987019185653 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1994480546304 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36265780 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10574723 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238043136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214103040 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1993570575872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80384065 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80410409 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚143465580905073 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _step‚140 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36265780 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36265782 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚146275694135808 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148269246708736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚148270174682112 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚1749 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21723204 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7884979400906324e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _runtime‚1997.871875112 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44118285 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚163547217508864 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚148269246708736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1994480546304 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32297321 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36265782 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80455426 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44169484 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36285940 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1995602019840 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1995602019840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148351987254784 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259001344 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36285940 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 trainer/global_step‚1749 _step‚141 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚148352915241472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚146357313221632 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80429056 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44169484 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚146357295218688 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1994692036096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32315145 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚161551054815232 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚146357313221632 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚143546640787459 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44143116 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚163638359433216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚621 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235061248 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚163638618434560 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32315766 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21735269 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚619 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10579876 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚143545715772895 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80429056 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚146357295218688 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1994692036096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚141558504328320 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1988136459139 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36285942 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚141558487288960 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1987228483935ç¼Ê€¨÷M MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _runtime‚1998.818238428 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚148352915241472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚146357313221632 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80429056 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44169484 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚146357295218688 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1994692036096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32315145 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚161551054815232 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚146357313221632 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚143546640787459 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44143116 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚163638359433216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚621 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235061248 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚163638618434560 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32315766 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21735269 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚619 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10579876 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚143545715772895 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80429056 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚146357295218688 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1994692036096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚141558504328320 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1988136459139 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36285942 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚141558487288960 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1987228483935 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36285942 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80455426 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4413ÕÝ%} 69484 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36285940 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1995602019840 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1995602019840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148351987254784 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259001344 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36285940 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 trainer/global_step‚1749 _step‚141 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚145430349421616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚150356630424064 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 trainer/global_step‚1799 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45389529 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252723200 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚150356630424064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45389529 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33193132 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10867315 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚152407216398336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2049658000896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚145430366460976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37273782 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165972114333184 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2041988465765 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37273780 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚150356648427008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10866678 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚168117489741824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 $ _timestamp‚1.788497994827199e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152406288424960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2049658000896 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228783104 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37273780 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚049ÔOò'·ìO D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚152407216398336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2049658000896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚145430366460976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37273782 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165972114333184 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2041988465765 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37273780 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚150356648427008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10866678 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚168117489741824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 $ _timestamp‚1.788497994827199e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152406288424960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2049658000896 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228783104 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37273780 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2050567971328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚150356648427008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚639 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚145430349421616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚150356630424064 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 trainer/global_step‚1799 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45389529 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252723200 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚150356630424064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45389529 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33193132 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10867315 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760‹þ7–2¸- _wandbruntime‚2053  _runtime‚2053º|ˆ»5¹òM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82681958 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235061248 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37293940 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22338521 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10873031 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45414386 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚152489956957696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 é´¸MÃ"D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚145 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚642 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚147552472755203 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33210910 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82708328 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚147553397769767 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2051689444864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2043105739043 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2051689444864 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37293942 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚166062363977728 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚152489028971008 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259001344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45388018 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚145509384055520 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2044013714247 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2050779461120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 trainer/global_step‚1799 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22338519 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82708328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚168208881540608 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33210910 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82708328 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚147553397769767 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2051689444864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2043105739043 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2051689444864 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37293942 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚166062363977728 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚152489028971008 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259001344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45388018 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚145509384055520 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2044013714247 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2050779461120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 trainer/global_step‚1799 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22338519 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82708328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚168208881540608 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37293940 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22338521 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10873031 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45414386 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚152489956957696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚145 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚642 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚147552472755203 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185‘ÿ‰®Ü2»- _wandbruntime‚2053  _runtime‚20530$vY¼Ó 1loss/SRqqt4/detection_loss‚2.3266514750087963 /loss/FRt1/detection_loss‚2.0755743412098484 /loss/FRt3/detection_loss‚1.7023761026064554 /loss/FBt3/detection_loss‚2.6700956523418427 % _timestamp‚1.7884979958453808e+09 /loss/FRt2/detection_loss‚2.8592479075305164 /loss/FRt3/assignment_loss‚1.741971804300944 2loss/SRqqt1/assignment_loss‚1.7195836926500003 .loss/FRt4/detection_loss‚3.280374190211296 1loss/SRqqt3/detection_loss‚0.9775629673124284 _step‚146 /loss/FBt4/assignment_loss‚0.392405577612093 /loss/FBt2/assignment_loss‚0.622970542863186 _runtime‚2053.622369016 trainer/global_step‚1799 /loss/FBt4/detection_loss‚2.4408241372842054 %loss/total_loss‚47.64706249403721 /loss/FRt1/assignment_loss‚3.100388174471648 0loss/FRt2/assignment_loss‚3.7840649373829365 2loss/SRqqt4/assignment_loss‚2.1939013467772916 1loss/FBt1/assignment_loss‚0.14622678973868297 .loss/FRt4/assignment_loss‚4.35368115901947 2loss/SRqqt2/assignment_loss‚2.0459754728909694 1loss/FBt3/assignment_loss‚0.10581432867826202 1loss/SRqqt1/detection_loss‚1.9938375246193674 0loss/SRqqt2/detection_loss‚2.385500276630575 /loss/FBt1/detection_loss‚0.9603152173467175 /loss/FBt2/detection_loss‚2.6180861070752144 epoch‚1 2loss/SRqqt3/assignment_loss‚1.1496327684744614¤dV7^½Ø /loss/FBt4/assignment_loss‚0.392405577612093 /loss/FBt2/assignment_loss‚0.622970542863186 _runtime‚2053.622369016 trainer/global_step‚1799 /loss/FBt4/detection_loss‚2.4408241372842054 %loss/total_loss‚47.64706249403721 /loss/FRt1/assignment_loss‚3.100388174471648 0loss/FRt2/assignment_loss‚3.7840649373829365 2loss/SRqqt4/assignment_loss‚2.1939013467772916 1loss/FBt1/assignment_loss‚0.14622678973868297 .loss/FRt4/assignment_loss‚4.35368115901947 2loss/SRqqt2/assignment_loss‚2.0459754728909694 1loss/FBt3/assignment_loss‚0.10581432867826202 1loss/SRqqt1/detection_loss‚1.9938375246193674 0loss/SRqqt2/detection_loss‚2.385500276630575 /loss/FBt1/detection_loss‚0.9603152173467175 /loss/FBt2/detection_loss‚2.6180861070752144 epoch‚1 2loss/SRqqt3/assignment_loss‚1.1496327684744614 1loss/SRqqt4/detection_loss‚2.3266514750087963 /loss/FRt1/detection_loss‚2.0755743412098484 /loss/FRt3/detection_loss‚1.7023761026064554 /loss/FBt3/detection_loss‚2.6700956523418427 % _timestamp‚1.7884979958453808e+09 /loss/FRt2/detection_loss‚2.8592479075305164 /loss/FRt3/assignment_loss‚1.741971804300944 2loss/SRqqt1/assignment_loss‚1.7195836926500003 .loss/FRt4/detection_loss‚3.280374190211296 1loss/SRqqt3/detection_loss‚0.9775629673124284 _step‚146’ø©“@“¾‚(:ˆ Í éÔÚ¼ám gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚152.899' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.22475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿yŠ¾b¿‚(:× Í éÔÚ‰úm disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sdb.out‚224.78173828125 disk.rbd2.out‚ 0.56640625 network.recv‚726076 disk.nvme0n1.out‚ 10900.8125 disk.nvme1n1.in‚ 9.0078125$ memory_percent‚99.96650218963623$ disk./.usageGB‚2994.578884124756 disk.sda.out‚224.78173828125 disk.dm-0.out‚23416.52734375 disk.sda2.in‚1.671875# disk.nvme1n1.out‚12515.71484375 disk.md0.in‚ 1.94140625 cpu‚25.156940704805084 disk.sdb2.in‚ 0.26953125! disk.sdb2.out‚224.78173828125 disk.sda.in‚1.671875# proc.memory.rssMB‚ 2992.33203125* proc.memory.percent‚18.263745307922363 network.sent‚ 3.365197e+06 disk.sdb1.in‚0 disk.md0p1.in‚ 1.94140625 disk.md0p1.out‚ 216.0546875 disk.md0.out‚ 216.0546875 disk.sda1.in‚0 disk.sdb.in‚ 0.26953125& proc.memory.availableMB‚ 5.48828125 disk.nvme0n1.in‚ 56.55859375 disk.rbd2.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚83.72967892875964 disk.sda1.out‚0 disk.rbd0.out‚ 0.279296875 disk.dm-0.in‚ 65.56640625! disk.sda2.out‚224.78173828125PÉ{Š“À‚(:ˆ Ü éÔÜÇól gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.763' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.94075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}}Ö^Á‚(:Ó Ü éÔ¯¯Ém" disk.nvme0n1.out‚ 10900.9765625# memory_percent‚99.9650239944458 disk.sdb2.out‚ 226.990234375 disk.sdb1.in‚0 disk.dm-0.in‚ 65.56640625 cpu‚25.155707270628895 disk.nvme1n1.in‚ 9.0078125# disk.nvme1n1.out‚12602.18359375 disk.sda.out‚ 226.990234375 network.sent‚ 3.461931e+06 disk.rbd2.in‚0 disk.sdb2.in‚ 0.26953125 disk.sdb1.out‚0 disk.md0p1.in‚ 1.94140625 disk.dm-0.out‚23503.16015625 disk.sdb.out‚ 226.990234375 disk.sda2.out‚ 226.990234375 disk.md0.out‚ 218.19921875 disk.rbd0.out‚ 0.279296875* proc.memory.percent‚18.263745307922363) disk./.usagePercent‚83.73903785715065$ disk./.usageGB‚2994.913604736328 disk.sda.in‚1.671875 disk.sda1.in‚0 disk.sdb.in‚ 0.26953125 disk.nvme0n1.in‚ 56.55859375 disk.rbd0.in‚0# proc.memory.rssMB‚ 2992.33203125 network.recv‚731768 disk.md0.in‚ 1.94140625 disk.md0p1.out‚ 218.19921875& proc.memory.availableMB‚ 5.73046875 proc.cpu.threads‚17 disk.rbd2.out‚ 0.56640625 disk.sda1.out‚0 disk.sda2.in‚1.671875!à0£“‚(:ˆ ë éÔµó‰m gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.891' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.72275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ“¡eÂ(:Ú ë éÔÍÔ¡m disk.dm-0.in‚ 65.56640625 disk.md0p1.in‚ 1.94140625$ memory_percent‚99.96497631072998* proc.memory.percent‚18.263745307922363 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd0.out‚ 0.279296875 network.sent‚ 3.465633e+06 disk.sdb.in‚ 0.26953125 disk.rbd2.in‚0! disk.sdb2.out‚228.52099609375 disk.sda.out‚228.52099609375 disk.md0p1.out‚ 219.66015625 network.recv‚735939$ disk./.usageGB‚2994.914726257324 disk.nvme0n1.in‚ 56.55859375 disk.rbd0.in‚0& proc.memory.availableMB‚ 5.73828125 disk.dm-0.out‚23715.68359375 disk.sda2.in‚1.671875! disk.nvme0n1.out‚ 10901.046875 disk.md0.in‚ 1.94140625 disk.sda1.out‚0 disk.sdb1.in‚0# proc.memory.rssMB‚ 2992.33203125 disk.sdb1.out‚0 disk.nvme1n1.in‚ 9.0078125( disk./.usagePercent‚83.7390692153471 disk.md0.out‚ 219.66015625 disk.rbd2.out‚ 0.56640625 cpu‚25.154608326418366 disk.sdb2.in‚ 0.26953125# disk.nvme1n1.out‚12814.63671875 disk.sdb.out‚228.52099609375! disk.sda2.out‚228.52099609375 disk.sda.in‚1.671875—¯ÇÛ’Ä‚(:‡ ú éÔðæÔm gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.266' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.3165 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢„´`Å‚(:Õ ú éÔÏùÇn disk.rbd0.in‚0 disk.md0p1.in‚ 1.94140625 cpu‚25.153485157941166 proc.cpu.threads‚17 network.sent‚ 3.471897e+06 disk.sda.in‚1.671875 disk.sda.out‚230.07275390625 disk.sdb1.in‚0 disk.sdb.out‚230.07275390625 disk.md0.in‚ 1.94140625* proc.memory.percent‚18.263745307922363 disk.sda2.in‚1.671875! disk.sda2.out‚230.07275390625 disk.nvme0n1.in‚ 56.55859375 disk.sda1.in‚0 disk.md0p1.out‚ 221.1484375) disk./.usagePercent‚83.76010853861077 disk.sdb.in‚ 0.26953125 disk.nvme1n1.in‚ 9.0078125 disk.sdb2.in‚ 0.26953125 disk.rbd0.out‚ 0.279296875$ memory_percent‚99.96497631072998$ disk./.usageGB‚2995.667194366455 disk.nvme0n1.out‚ 10901.1875" disk.nvme1n1.out‚ 13004.3515625 disk.sda1.out‚0 disk.sdb1.out‚0 disk.dm-0.out‚ 23905.5390625# proc.memory.rssMB‚ 2992.33203125 network.recv‚740768 disk.rbd2.in‚0 disk.rbd2.out‚ 0.56640625! disk.sdb2.out‚230.07275390625 disk.md0.out‚ 221.1484375 disk.dm-0.in‚ 65.56640625& proc.memory.availableMB‚ 5.73828125´·à‰2Æ- _wandbruntime‚2110  _runtime‚2110è°ÎÄǾ _runtime‚2110.954722575 _step‚147 % lr-AdamW/pg1‚0.001043396216291913 % lr-AdamW/pg2‚0.001043396216291913 trainer/global_step‚1849 $ _timestamp‚1.788498053177426e+09Ÿ‰ôÓÉÈà _step‚147 % lr-AdamW/pg1‚0.001043396216291913 % lr-AdamW/pg2‚0.001043396216291913 trainer/global_step‚1849 $ _timestamp‚1.788498053177426e+09 _runtime‚2110.954722575“Qm22É- _wandbruntime‚2110  _runtime‚2110ªceÊèO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚148 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38281780 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚156544258114560 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34088190 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚154437602718208 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚170483423495680 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2105745425920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38281782 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38281782 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38281780 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248528896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2106655396352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚172687533286400 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 _runtime‚2110.955553949 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156543330141184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚617 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34087573 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84889869 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚849J$(Ö16213 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2105745425920 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚151480019884045 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 % _timestamp‚1.7884980531781962e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚156543330141184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11158484 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2203885201920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38281782 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38281782 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38281780 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248528896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2106655396352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚172687533286400 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 _runtime‚2110.955553949 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156543330141184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚617 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34087573 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84889869 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84916213 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2105745425920 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚151480019884045 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 % _timestamp‚1.7884980531781962e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚156543330141184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11158484 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2203885201920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚156544258114560 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34088190 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚154437602718208 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚170483423495680 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570”‰¦ÿ2Ì- _wandbruntime‚2111  _runtime‚2111y½„-ø&ÍòM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156626998673920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7884980542156444e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38301942 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240126976 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2099890969355 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34105981 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚170573625260032 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚151559229737511 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34105366 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11164210 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46632920 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84961230 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22941771 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216186880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2205285801472 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2098982994151 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 _runtime‚2111.996915601 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11163597 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚172778694874624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46659288 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚615 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚149460263782720 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚156626998673920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚172778935001600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚154519221804032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚154519203801088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22941769 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46659288 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46632920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38301940 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2106866886144 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84934860 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38301940 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚154519221804032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚170573649200128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 trainer/global_step‚1849 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38301942LG?êÎ÷M E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38301940 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚154519221804032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚170573649200128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 trainer/global_step‚1849 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38301942 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156626998673920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7884980542156444e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38301942 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240126976 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2099890969355 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34105981 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚170573625260032 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚151559229737511 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34105366 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11164210 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46632920 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84961230 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22941771 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216186880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2205285801472 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2098982994151 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 _runtime‚2111.996915601 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11163597 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚172778694874624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46659288 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚615 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_¥ÜÐ pool.allocated‚149460263782720 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚156626998673920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚172778935001600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚154519221804032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚154519203801088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22941769 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46659288 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46632920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38301940 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2106866886144 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84934860•iþ?µ2Ï- _wandbruntime‚2111  _runtime‚2111g³ëøYÐÓ 1loss/FBt2/assignment_loss‚0.21582388652902107 /loss/FRt4/detection_loss‚3.3806790991833338 /loss/FBt3/detection_loss‚2.8368786692619326 %loss/total_loss‚47.72993512949432 _runtime‚2111.998268757 0loss/FRt1/assignment_loss‚2.8044909441724735 /loss/FRt4/assignment_loss‚4.226463418257864 2loss/SRqqt1/assignment_loss‚1.2864788247538463 2loss/SRqqt2/assignment_loss‚1.9401071798056364 .loss/FRt3/detection_loss‚1.863243296471509 /loss/FRt2/assignment_loss‚4.006125563383103 0loss/FRt3/assignment_loss‚1.8658816348422658 1loss/SRqqt1/detection_loss‚2.1448678706373485 .loss/FBt4/detection_loss‚2.749128724847521 _step‚150 0loss/FBt4/assignment_loss‚0.3258447084188317 .loss/FRt1/detection_loss‚2.033615814941592 /loss/FRt2/detection_loss‚2.5465008918195964 0loss/SRqqt3/detection_loss‚1.039240375024463 trainer/global_step‚1849 % _timestamp‚1.7884980542210805e+09 2loss/SRqqt3/assignment_loss‚1.1143570301985932 1loss/SRqqt2/detection_loss‚2.7020272708364894 0loss/SRqqt4/detection_loss‚2.676744302113851 epoch‚1 1loss/FBt1/assignment_loss‚0.20813015068227392 0loss/FBt3/assignment_loss‚0.1896177028887905 .loss/FBt1/detection_loss‚0.908709860551052 /loss/FBt2/detection_loss‚2.3698954502741496 2loss/SRqqt4/assignment_loss‚2.2950824595987798"/n^ÑØ .loss/FBt4/detection_loss‚2.749128724847521 _step‚150 0loss/FBt4/assignment_loss‚0.3258447084188317 .loss/FRt1/detection_loss‚2.033615814941592 /loss/FRt2/detection_loss‚2.5465008918195964 0loss/SRqqt3/detection_loss‚1.039240375024463 trainer/global_step‚1849 % _timestamp‚1.7884980542210805e+09 2loss/SRqqt3/assignment_loss‚1.1143570301985932 1loss/SRqqt2/detection_loss‚2.7020272708364894 0loss/SRqqt4/detection_loss‚2.676744302113851 epoch‚1 1loss/FBt1/assignment_loss‚0.20813015068227392 0loss/FBt3/assignment_loss‚0.1896177028887905 .loss/FBt1/detection_loss‚0.908709860551052 /loss/FBt2/detection_loss‚2.3698954502741496 2loss/SRqqt4/assignment_loss‚2.2950824595987798 1loss/FBt2/assignment_loss‚0.21582388652902107 /loss/FRt4/detection_loss‚3.3806790991833338 /loss/FBt3/detection_loss‚2.8368786692619326 %loss/total_loss‚47.72993512949432 _runtime‚2111.998268757 0loss/FRt1/assignment_loss‚2.8044909441724735 /loss/FRt4/assignment_loss‚4.226463418257864 2loss/SRqqt1/assignment_loss‚1.2864788247538463 2loss/SRqqt2/assignment_loss‚1.9401071798056364 .loss/FRt3/detection_loss‚1.863243296471509 /loss/FRt2/assignment_loss‚4.006125563383103 0loss/FRt3/assignment_loss‚1.8658816348422658 1loss/SRqqt1/detection_loss‚2.1448678706373485– ]v<’Ò‚(:‡ ‰¡éÔŸºèl gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.434' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.8585 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\žTdZÓ‚(:Ï ‰¡éÔºõŽn disk.md0p1.in‚ 1.94140625 disk.dm-0.in‚ 65.56640625 disk.dm-0.out‚24173.60546875 disk.md0.in‚ 1.94140625 proc.cpu.threads‚17% disk./.usageGB‚2995.6643295288086 disk.sdb1.out‚0& proc.memory.availableMB‚ 5.94140625 disk.rbd2.out‚ 0.56640625 disk.sdb2.in‚ 0.26953125$ memory_percent‚99.96373653411865 disk.rbd0.in‚0 disk.sdb.in‚ 0.26953125 disk.sdb2.out‚231.8017578125 disk.sda.in‚1.671875 disk.sda.out‚231.8017578125 cpu‚25.15253698482472 network.recv‚744090) disk./.usagePercent‚83.76002843655115 disk.sdb.out‚231.8017578125 disk.nvme1n1.in‚ 9.0078125# disk.nvme1n1.out‚13272.33203125 disk.sda2.in‚1.671875 disk.nvme0n1.in‚ 56.55859375 disk.md0.out‚222.8125 disk.sda1.in‚0 disk.sdb1.in‚0 disk.md0p1.out‚222.8125 disk.sda2.out‚231.8017578125* proc.memory.percent‚18.263745307922363 disk.sda1.out‚0" disk.nvme0n1.out‚ 10901.2734375 disk.rbd2.in‚0# proc.memory.rssMB‚ 2992.33203125 network.sent‚ 3.49003e+06 disk.rbd0.out‚ 0.279296875ÑåQl“Ô‚(:ˆ ˜¡éÔ“ö­m gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.903' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.72575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±¥Žr`Õ‚(:Õ ˜¡éÔÙÉm disk.rbd0.out‚ 0.279296875 disk.dm-0.in‚ 65.56640625 disk.md0.out‚ 224.1953125 disk.sda.in‚1.671875 disk.sda2.in‚1.671875" disk.nvme0n1.out‚ 10901.4765625 disk.nvme1n1.in‚ 9.0078125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sdb2.in‚ 0.26953125 disk.sda1.out‚0# proc.memory.rssMB‚ 2992.33203125* proc.memory.percent‚18.263745307922363 proc.cpu.threads‚17 disk.md0.in‚ 1.94140625% proc.memory.availableMB‚ 6.1171875 network.recv‚750036 disk.nvme0n1.in‚ 56.55859375! disk.nvme1n1.out‚ 13476.828125 disk.rbd2.in‚0! disk.sdb2.out‚233.23388671875 disk.sda.out‚233.23388671875# memory_percent‚99.9626636505127 cpu‚25.15137284921927 network.sent‚ 3.586747e+06 disk.sdb1.out‚0 disk.md0p1.in‚ 1.94140625% disk./.usageGB‚2995.9143104553223 disk.sdb1.in‚0 disk.md0p1.out‚ 224.1953125 disk.dm-0.out‚ 24378.3046875 disk.sdb.in‚ 0.26953125 disk.sdb.out‚233.23388671875! disk.sda2.out‚233.23388671875 disk.rbd2.out‚ 0.56640625) disk./.usagePercent‚83.76701800788162¯nß…’Ö‚(:‡ §¡éÔÕÄm gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°ìÿäbׂ(:× §¡éÔÉŸ´n% disk./.usageGB‚2995.9143104553223! disk.sda2.out‚234.86181640625 disk.md0p1.out‚ 225.76171875$ memory_percent‚99.96414184570312 disk.md0p1.in‚ 1.94140625# disk.nvme1n1.out‚13612.66015625! disk.sdb2.out‚234.86181640625 cpu‚25.150421616970824 disk.sda.in‚1.671875 disk.dm-0.in‚ 65.56640625 disk.rbd2.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2992.33203125 disk.md0.in‚ 1.94140625 disk.rbd2.out‚ 0.56640625 network.recv‚754199 disk.rbd0.out‚ 0.279296875 disk.dm-0.out‚24514.20703125* proc.memory.percent‚18.263745307922363 disk.sda.out‚234.86181640625 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sdb.in‚ 0.26953125 disk.nvme0n1.in‚ 56.55859375 disk.nvme1n1.in‚ 9.0078125 disk.sdb2.in‚ 0.26953125 disk.sda1.in‚0 disk.sdb1.in‚0 disk.sdb.out‚234.86181640625 disk.sda2.in‚1.671875! disk.nvme0n1.out‚ 10901.546875 disk.md0.out‚ 225.76171875! proc.memory.availableMB‚5.875 proc.cpu.threads‚17 network.sent‚ 3.590348e+06) disk./.usagePercent‚83.76701800788162ß±›“Ø‚(:ˆ ¶¡éÔØ›Ýl gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾ EXÙ‚(:Í ¶¡éÔô¨’m disk.sda2.out‚236.4306640625! disk.nvme0n1.out‚ 10901.796875 disk.rbd2.in‚0# proc.memory.rssMB‚ 2992.33203125 proc.cpu.threads‚17$ disk./.usageGB‚2996.039337158203 disk.sdb2.out‚236.4306640625 disk.sda1.in‚0 disk.nvme1n1.out‚ 14081.84375 disk.md0.in‚ 1.94140625 disk.md0p1.out‚ 227.265625 disk.sdb.in‚ 0.26953125 disk.md0.out‚ 227.265625 disk.dm-0.out‚ 24983.640625 disk.nvme1n1.in‚ 9.0078125& proc.memory.availableMB‚ 6.08984375 network.recv‚759069( disk./.usagePercent‚83.7705138068219 disk.sdb2.in‚ 0.26953125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚ 1.94140625 disk.nvme0n1.in‚ 56.55859375 disk.rbd2.out‚ 0.56640625 disk.sda.in‚1.671875 disk.sda.out‚236.4306640625 disk.rbd0.in‚0 disk.rbd0.out‚ 0.279296875 disk.dm-0.in‚ 65.56640625 disk.sdb.out‚236.4306640625* proc.memory.percent‚18.263745307922363 cpu‚25.149419580978527$ memory_percent‚99.96283054351807 network.sent‚ 3.598956e+06 disk.sda1.out‚0 disk.sda2.in‚1.671875—õG2Ú- _wandbruntime‚2169  _runtime‚2169ËNžúÆÛÀ $ _timestamp‚1.788498111692825e+09 _runtime‚2169.470385872 _step‚151 & lr-AdamW/pg1‚0.0010419467432433408 & lr-AdamW/pg2‚0.0010419467432433408 trainer/global_step‚1899‘ÌLÆËÜÅ & lr-AdamW/pg1‚0.0010419467432433408 & lr-AdamW/pg2‚0.0010419467432433408 trainer/global_step‚1899 $ _timestamp‚1.788498111692825e+09 _runtime‚2169.470385872 _step‚151—l4ç]2Ý- _wandbruntime‚2169  _runtime‚2169{bésî'ÞèO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23532956 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39289782 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚158518557009408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11450103 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34983059 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚158518539006464 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚160681299830784 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2262943715840 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2161832850944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34982430 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2162742821376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39289780 trainer/global_step‚1899 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158518539006464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚177257412293632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2262727515648 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47879333 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚155486776866353 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚160681299830784 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47852991 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160680371857408 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47852991 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47879333 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 % _timestamp‚1.7884981116939414e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23532954 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚627 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚158518557009408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216200192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚153332125915376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _step‚152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39289782 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2162742821376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39289780 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2161832850944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚160680371857408 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2154650950977 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚153332108876016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177257652433920 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚155485851851997 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2153742975981 _runtime‚2169.471324206 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1ź°`å%ßíO A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚153332125915376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _step‚152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39289782 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2162742821376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39289780 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2161832850944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚160680371857408 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2154650950977 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚153332108876016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177257652433920 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚155485851851997 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2153742975981 _runtime‚2169.471324206 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23532956 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39289782 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚158518557009408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11450103 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34983059 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚158518539006464 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚160681299830784 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2262943715840 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2161832850944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34982430 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2162742821376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39289780 trainer/global_step‚1899 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158518539006464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚177257412293632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2262727515648 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47879333 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚155486776866353 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚160681299830784 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47852991 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160680371857408 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47852991 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47879333 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 % _timestamp‚1.7884981116939414e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23532954 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚627 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚158518557009408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216200192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0˜§E»V2à- _wandbruntime‚2170  _runtime‚2170‚l ø&áòM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39309940 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2163864294912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23545019 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39309942 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35000911 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚153411126470560 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87214132 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35000270 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚155565986719819 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23545021 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11455251 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 _runtime‚2170.621089048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39309942 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177348796262912 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2264116647424 _step‚153 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚158600158092288 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚155566911734383 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2263885780480 trainer/global_step‚1899¹åŠ¦ý&â÷M _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚175084910482432 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚153411143509920 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2154860249259 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47904190 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚175084934422528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2155768224463 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35000270 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚155565986719819 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23545021 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11455251 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 _runtime‚2170.621089048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39309942 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177348796262912 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2264116647424 _step‚153 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚158600158092288 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚155566911734383 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2263885780480 trainer/global_step‚1899 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39309940 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2163864294912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23545019 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39309942 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35000911 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚153411126470560 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87214132 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11741307 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162599511300608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚600 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚181827414758400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 207811584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40297780 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24136206 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11740707 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35877513 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164817413573632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2217920275968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚162599493297664 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚179505993940480 A8DeviceStatsßE0>Monitor.on_train_batch_start/num_device_alloc‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚602 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚159492608834305 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2210528206085 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162599511300608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚159493533848661 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162599493297664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164817413573632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2217920275968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚162599493297664 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚179505993940480 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚602 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚159492608834305 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2210528206085 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162599511300608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚159493533848661 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162599493297664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11741307 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162599511300608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432œ\S=c2ô- _wandbruntime‚2223  _runtime‚2223fø&õòM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89467034 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚164900154119680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2219041736192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚603 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162681112383488 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 % _timestamp‚1.7884981658251853e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89440664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2211645479571 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49122724 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2210737504367 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248515584 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89440664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚179596195704832 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2322622008832 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49149092 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚159572743702127 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164901082106368 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚601 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2219041736192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚159573668716691 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40317940 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24148269 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49149092 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2219951719936 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚157362023237120 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 trainer/global_step‚1949 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40317942 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚181918817713664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚162681112383488 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89467034 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35894796 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2322846584320 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40317940 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162681130386432 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2219951719936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11746527 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40317942 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚601 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2219041736192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚159573668716691 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40317940 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24148269 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49149092 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2219951719936 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚157362023237120 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 trainer/global_step‚1949 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40317942 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚181918817713664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚162681112383488 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89467034 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35894796 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2322846584320 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40317940 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162681130386432 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2219951719936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11746527 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40317942 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89467034 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚164900154119680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2219041736192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚603 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162681112383488 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 % _timestamp‚1.7884981658251853e+09($¤)2÷- _wandbruntime‚2223  _runtime‚2223÷1´²YøÓ 1loss/FBt3/assignment_loss‚0.15203628500489685 .loss/FRt2/detection_loss‚2.716792258314597 0loss/SRqqt1/detection_loss‚2.337439074118932 epoch‚1 trainer/global_step‚1949 0loss/FRt3/assignment_loss‚2.0291398363454003 /loss/FRt4/assignment_loss‚3.649140775203705 1loss/SRqqt2/assignment_loss‚1.789317427788462 0loss/FBt1/assignment_loss‚0.2409643572237638 .loss/FRt4/detection_loss‚2.949157146116098 /loss/FBt1/detection_loss‚0.9780336898069576 /loss/FBt2/detection_loss‚3.4407834659020105 1loss/SRqqt1/assignment_loss‚1.442813848455747 .loss/FRt1/detection_loss‚1.930106258692427 0loss/SRqqt3/detection_loss‚1.123142848689375 _runtime‚2223.605258058 2loss/SRqqt3/assignment_loss‚1.3033536248840392 1loss/FBt2/assignment_loss‚0.41448004448814874 0loss/FRt1/assignment_loss‚2.6623041921191746 0loss/FRt2/assignment_loss‚3.4671858151753745 1loss/FBt4/assignment_loss‚0.17734166994029824 .loss/FRt3/detection_loss‚1.768952556273767 /loss/FBt3/detection_loss‚3.1141304799488614 % _timestamp‚1.7884981658281906e+09 2loss/SRqqt4/assignment_loss‚1.6197968044628699 /loss/FBt4/detection_loss‚2.1244775312287465 1loss/SRqqt2/detection_loss‚2.4765935591288977 1loss/SRqqt4/detection_loss‚2.6765501499176025 %loss/total_loss‚46.58403369923015 _step‚158ÕÃ…É^ùØ 0loss/FRt1/assignment_loss‚2.6623041921191746 0loss/FRt2/assignment_loss‚3.4671858151753745 1loss/FBt4/assignment_loss‚0.17734166994029824 .loss/FRt3/detection_loss‚1.768952556273767 /loss/FBt3/detection_loss‚3.1141304799488614 % _timestamp‚1.7884981658281906e+09 2loss/SRqqt4/assignment_loss‚1.6197968044628699 /loss/FBt4/detection_loss‚2.1244775312287465 1loss/SRqqt2/detection_loss‚2.4765935591288977 1loss/SRqqt4/detection_loss‚2.6765501499176025 %loss/total_loss‚46.58403369923015 _step‚158 1loss/FBt3/assignment_loss‚0.15203628500489685 .loss/FRt2/detection_loss‚2.716792258314597 0loss/SRqqt1/detection_loss‚2.337439074118932 epoch‚1 trainer/global_step‚1949 0loss/FRt3/assignment_loss‚2.0291398363454003 /loss/FRt4/assignment_loss‚3.649140775203705 1loss/SRqqt2/assignment_loss‚1.789317427788462 0loss/FBt1/assignment_loss‚0.2409643572237638 .loss/FRt4/detection_loss‚2.949157146116098 /loss/FBt1/detection_loss‚0.9780336898069576 /loss/FBt2/detection_loss‚3.4407834659020105 1loss/SRqqt1/assignment_loss‚1.442813848455747 .loss/FRt1/detection_loss‚1.930106258692427 0loss/SRqqt3/detection_loss‚1.123142848689375 _runtime‚2223.605258058 2loss/SRqqt3/assignment_loss‚1.3033536248840392 1loss/FBt2/assignment_loss‚0.41448004448814874ž¶\Jãaú‚(:Ö ¢éÔœÿÎo disk.sda1.out‚0 disk.sdb1.in‚0# proc.memory.rssMB‚ 2992.33203125 proc.cpu.threads‚17 disk.sdb.in‚ 0.26953125 disk.sdb.out‚244.83349609375 disk.nvme1n1.out‚ 14711.09375 disk.md0.out‚ 235.33203125 disk.md0p1.in‚ 1.94140625* proc.memory.percent‚18.263745307922363 network.sent‚ 3.73873e+06 network.recv‚780410! disk.sda2.out‚244.83349609375" disk.nvme0n1.out‚ 10902.9453125 disk.dm-0.out‚ 25614.0390625 disk.nvme0n1.in‚ 56.55859375 disk.dm-0.in‚ 65.56640625 disk.sda2.in‚1.671875 disk.sda.out‚244.83349609375 disk.rbd0.in‚0 cpu‚25.144735827594456 disk.rbd2.out‚ 0.56640625! disk.sdb2.out‚244.83349609375 disk.sda.in‚1.671875 disk.sda1.in‚0 disk.rbd0.out‚ 0.279296875 disk.md0p1.out‚ 235.33203125$ memory_percent‚99.96495246887207% proc.memory.availableMB‚ 5.7421875 disk.sdb1.out‚0 disk.nvme1n1.in‚ 9.0078125) disk./.usagePercent‚83.72804595601285 disk.md0.in‚ 1.94140625 disk.rbd2.in‚0 disk.sdb2.in‚ 0.26953125$ disk./.usageGB‚2994.520481109619•¾å’û‚(:‡ ¢éÔõ–ào gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚118.142' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.5355 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒâ1å“ü‚(:ˆ ¢éÔúòûk gpu.0.gpu‚26.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚142.313' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.57825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»n# `ý‚(:Õ ¢éÔÉ‘êl disk.md0.out‚ 236.72265625 network.recv‚786183 disk.sda1.in‚0 disk.sdb1.out‚0 disk.sda2.out‚246.2900390625# disk.nvme1n1.out‚14887.38671875* proc.memory.percent‚18.263745307922363 network.sent‚ 3.8332e+06) disk./.usagePercent‚83.73700373414256 disk.sda1.out‚0 disk.rbd0.out‚ 0.279296875 disk.sdb1.in‚0 disk.md0p1.out‚ 236.72265625 disk.md0.in‚ 1.94140625 disk.nvme1n1.in‚ 9.0078125 disk.rbd2.out‚ 0.56640625 disk.sdb2.out‚246.2900390625 disk.rbd0.in‚0 disk.dm-0.out‚25790.49609375 disk.sda2.in‚1.671875$ memory_percent‚99.96657371520996 proc.cpu.threads‚17% disk./.usageGB‚2994.8408546447754 disk.sdb2.in‚ 0.26953125 disk.sdb.out‚246.2900390625 disk.rbd2.in‚0 cpu‚25.143846168806373 disk.sda.in‚1.671875 disk.sdb.in‚ 0.26953125! disk.nvme0n1.out‚ 10903.109375 disk.sda.out‚246.2900390625 disk.md0p1.in‚ 1.94140625 disk.dm-0.in‚ 65.56640625 disk.nvme0n1.in‚ 56.55859375% proc.memory.availableMB‚ 5.4765625# proc.memory.rssMB‚ 2992.33203125i¸ôÄ“þ‚(:ˆ Ÿ¢éÔ„Àÿk gpu.0.gpu‚54.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚177.071' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.26775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñtØÇZÿ‚(:Ï Ÿ¢éÔ±µ­m disk.sda.in‚1.671875 disk.dm-0.out‚25921.30078125( disk./.usagePercent‚83.7389400494427 disk.sdb2.out‚ 247.76171875 disk.rbd0.in‚0$ memory_percent‚99.96504783630371 disk.md0.in‚ 1.94140625 disk.rbd2.out‚ 0.56640625 disk.md0p1.in‚ 1.94140625 disk.sda2.in‚1.671875% proc.memory.availableMB‚ 5.7265625# proc.memory.rssMB‚ 2992.33203125% disk./.usageGB‚2994.9101066589355 disk.nvme1n1.in‚ 9.0078125 disk.sdb1.in‚0 disk.sdb.out‚ 247.76171875 cpu‚25.143219680272466 proc.cpu.threads‚17 disk.sdb2.in‚ 0.26953125 disk.rbd0.out‚ 0.279296875 disk.md0p1.out‚ 238.13671875 disk.dm-0.in‚ 65.56640625 disk.sdb.in‚ 0.26953125" disk.nvme0n1.out‚ 10903.2109375 disk.md0.out‚ 238.13671875 disk.sda.out‚ 247.76171875 disk.sda2.out‚ 247.76171875* proc.memory.percent‚18.263745307922363 disk.sda1.out‚0 network.sent‚ 3.839034e+06 network.recv‚790363# disk.nvme1n1.out‚15018.08984375 disk.sda1.in‚0 disk.sdb1.out‚0 disk.nvme0n1.in‚ 56.55859375 disk.rbd2.in‚0U82€- _wandbruntime‚2275  _runtime‚2275ö±Ÿ¼ÇÁ & lr-AdamW/pg2‚0.0010387256646070251 trainer/global_step‚1999 % _timestamp‚1.7884982174855063e+09 _runtime‚2275.262821903 _step‚159 & lr-AdamW/pg1‚0.0010387256646070251Ê*°”Ì‚Æ & lr-AdamW/pg2‚0.0010387256646070251 trainer/global_step‚1999 % _timestamp‚1.7884982174855063e+09 _runtime‚2275.262821903 _step‚159 & lr-AdamW/pg1‚0.0010387256646070251Ÿ!ç‰r2ƒ- _wandbruntime‚2275  _runtime‚2275û¨?î'„èO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36771786 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50342795 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚186397684478464 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41305780 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚161233885369776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2274007700992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168955383263232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168954455289856 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 _step‚160 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2274917671424 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2266405461193 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91648575 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24739456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224588800 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184017279162880 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 trainer/global_step‚1999 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚618 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36771168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12032330 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2274917671424 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168954455289856 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50369137 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚168955383263232 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚163499365816613 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12031714 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248528896 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚618 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36771168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12032330 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2274917671424 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168954455289856 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50369137 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚168955383263232 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚163499365816613 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12031714 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248528896 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36771786 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50342795 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚186397684478464 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41305780 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚161233885369776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2274007700992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168955383263232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168954455289856 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 _step‚160 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2274917671424 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2266405461193 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91648575 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24739456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224588800 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184017279162880 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 trainer/global_step‚1999 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚186488843713536 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 _runtime‚2276.214426121 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41325942 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12038057 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚184107504867328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12037455 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚161312885924960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚169038123822592 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41325942 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36789578 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚161312902964320 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50393994 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚169038123822592 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚163579500684435 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2276039144960 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2275129161216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚184107480927232 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2381581070336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚169037195835904 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41325940 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2276039144960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50367626 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36788974 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2267522734679 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚602 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚166762084677632 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91719936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2381362786304 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218284032 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚163580425698999 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91693566 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91693566 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚604 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚166762084677632šPÕ’ü&ˆöM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2381581070336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚169037195835904 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41325940 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2276039144960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50367626 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36788974 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2267522734679 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚602 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚166762084677632 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91719936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2381362786304 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218284032 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚163580425698999 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91693566 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91693566 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚604 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚166762084677632 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚166762066674688 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24751521 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91719936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2275129161216 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50367626 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242224128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚186488843713536 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 _runtime‚2276.214426121 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41325942 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12038057 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚184107504867328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12037455 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚161312885924960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚169038123822592 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41325942 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36789578 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚161312902964320 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50393994 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚169038123822592 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚163579500684435 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2276039144960 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2275129161216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚184107480927232¡(Î 2‰- _wandbruntime‚2276  _runtime‚2276´N¾TŠÎ 1loss/SRqqt1/detection_loss‚2.5546307403307695 epoch‚1 2loss/SRqqt1/assignment_loss‚1.6688762662502437 /loss/FRt3/detection_loss‚2.1003906288925482 /loss/FBt1/detection_loss‚0.7677494910888868 -loss/FBt3/detection_loss‚2.91071879863739 /loss/FRt2/detection_loss‚2.1834361118321515 $ _timestamp‚1.788498218438448e+09 0loss/FBt2/assignment_loss‚0.3435379965654243 1loss/FBt3/assignment_loss‚0.18641006667166948 1loss/SRqqt4/detection_loss‚2.7079060863364828 /loss/FRt4/assignment_loss‚5.376821744441986 1loss/FBt1/assignment_loss‚0.15800793041843003 /loss/FRt1/detection_loss‚1.8128818487098743 .loss/FBt4/detection_loss‚2.641821230159086 _step‚162 %loss/total_loss‚48.19179204144348 0loss/FRt3/assignment_loss‚1.7213990165262807 0loss/SRqqt4/assignment_loss‚1.92583153413778 1loss/FBt4/assignment_loss‚0.24274617787592861 /loss/FRt4/detection_loss‚3.6472138583660128 /loss/SRqqt2/detection_loss‚2.27791956932314 _runtime‚2276.215443456 /loss/FRt1/assignment_loss‚2.840089275393375 0loss/FRt2/assignment_loss‚3.7533101524625505 1loss/SRqqt2/assignment_loss‚1.806558351362905 2loss/SRqqt3/assignment_loss‚1.1273642134246895 /loss/SRqqt3/detection_loss‚1.20794577404712 .loss/FBt2/detection_loss‚2.228225178188748 trainer/global_step‚1999Õ<ªMY‹Ó 0loss/SRqqt4/assignment_loss‚1.92583153413778 1loss/FBt4/assignment_loss‚0.24274617787592861 /loss/FRt4/detection_loss‚3.6472138583660128 /loss/SRqqt2/detection_loss‚2.27791956932314 _runtime‚2276.215443456 /loss/FRt1/assignment_loss‚2.840089275393375 0loss/FRt2/assignment_loss‚3.7533101524625505 1loss/SRqqt2/assignment_loss‚1.806558351362905 2loss/SRqqt3/assignment_loss‚1.1273642134246895 /loss/SRqqt3/detection_loss‚1.20794577404712 .loss/FBt2/detection_loss‚2.228225178188748 trainer/global_step‚1999 1loss/SRqqt1/detection_loss‚2.5546307403307695 epoch‚1 2loss/SRqqt1/assignment_loss‚1.6688762662502437 /loss/FRt3/detection_loss‚2.1003906288925482 /loss/FBt1/detection_loss‚0.7677494910888868 -loss/FBt3/detection_loss‚2.91071879863739 /loss/FRt2/detection_loss‚2.1834361118321515 $ _timestamp‚1.788498218438448e+09 0loss/FBt2/assignment_loss‚0.3435379965654243 1loss/FBt3/assignment_loss‚0.18641006667166948 1loss/SRqqt4/detection_loss‚2.7079060863364828 /loss/FRt4/assignment_loss‚5.376821744441986 1loss/FBt1/assignment_loss‚0.15800793041843003 /loss/FRt1/detection_loss‚1.8128818487098743 .loss/FBt4/detection_loss‚2.641821230159086 _step‚162 %loss/total_loss‚48.19179204144348 0loss/FRt3/assignment_loss‚1.7213990165262807¢CEõ4Œ‚(:„ ®¢éÔªòªl gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚178.02' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.505 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙãDôb‚(:× ®¢éÔšÜÛl disk.sda1.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚ 1.94140625 disk.sdb.in‚ 0.26953125! disk.sda2.out‚249.64111328125 proc.cpu.threads‚17# disk.nvme1n1.out‚15153.25390625 disk.sda.out‚249.64111328125 disk.sda1.out‚0 disk.rbd0.out‚ 0.294921875 disk.md0p1.out‚ 239.94921875 network.sent‚ 3.8454e+06% disk./.usageGB‚2994.9143829345703 disk.sdb.out‚249.64111328125$ memory_percent‚99.96371269226074 disk.nvme1n1.in‚ 9.0078125 disk.sdb1.in‚0 disk.dm-0.out‚26056.98046875) disk./.usagePercent‚83.73905961589921 disk.rbd2.out‚0.578125 disk.rbd0.in‚0 disk.dm-0.in‚ 65.56640625 disk.sda2.in‚1.671875 disk.nvme0n1.in‚ 56.55859375 disk.md0.in‚ 1.94140625" disk.nvme0n1.out‚ 10903.7265625% proc.memory.availableMB‚ 5.9453125 disk.md0.out‚ 239.94921875 disk.rbd2.in‚0 disk.sdb2.in‚ 0.26953125 disk.sda.in‚1.671875# proc.memory.rssMB‚ 2992.33203125* proc.memory.percent‚18.263745307922363 cpu‚25.14241056287251 network.recv‚794998! disk.sdb2.out‚249.64111328125âÑè“Ž‚(:ˆ ½¢éÔä³Ök gpu.0.gpu‚27.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚178.449' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.61225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þä7N‚(:à ½¢éÔܶðl disk.dm-0.out‚ 26272.2890625( disk./.usagePercent‚83.7425604278845" disk.nvme1n1.out‚ 15368.4765625 disk.rbd2.in‚0! disk.sdb2.out‚251.26904296875% disk./.usageGB‚2995.0395889282227 disk.sdb2.in‚ 0.26953125 disk.sda.in‚1.671875 disk.nvme0n1.in‚ 56.55859375" proc.memory.availableMB‚5.4375 disk.sda1.out‚0 disk.md0p1.out‚241.5 disk.md0.in‚ 1.94140625 disk.md0p1.in‚ 1.94140625! disk.sda2.out‚251.26904296875 disk.sda.out‚251.26904296875 disk.sda2.in‚1.671875 disk.nvme0n1.out‚ 10903.8125# proc.memory.rssMB‚ 2992.33203125 network.sent‚ 3.952221e+06 disk.rbd2.out‚0.640625 disk.rbd0.in‚0 disk.rbd0.out‚ 0.294921875 disk.sdb.in‚ 0.26953125 disk.sdb.out‚251.26904296875* proc.memory.percent‚18.263745307922363 cpu‚25.141809544278246 network.recv‚800088 disk.md0.out‚241.5 disk.sda1.in‚0 disk.sdb1.out‚0$ memory_percent‚99.96681213378906 proc.cpu.threads‚17 disk.nvme1n1.in‚ 9.0078125 disk.sdb1.in‚0 disk.dm-0.in‚ 65.56640625 lZ@“‚(:ˆ Ì¢éÔ´ÞÑk gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚182.031' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.50775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L8ÁIW‘‚(:Ì Ì¢éÔÓÖÂl disk.sda1.in‚0 disk.nvme1n1.out‚ 15580.71875 network.sent‚ 3.958251e+06 disk.sdb2.out‚ 252.556640625 disk.dm-0.in‚ 65.56640625 disk.sdb.out‚ 252.556640625 disk.nvme0n1.in‚ 56.55859375" disk.nvme0n1.out‚ 10903.9140625) disk./.usagePercent‚83.73122571979647 disk.rbd0.out‚ 0.357421875 disk.sdb1.in‚0 disk.sda2.in‚1.671875 disk.md0.in‚ 1.94140625$ memory_percent‚99.96523857116699 proc.cpu.threads‚17 disk.md0p1.out‚ 242.72265625 disk.sdb1.out‚0 disk.md0p1.in‚ 1.94140625 disk.dm-0.out‚ 26484.6328125 disk.nvme1n1.in‚ 9.0078125 disk.md0.out‚ 242.72265625* proc.memory.percent‚18.263745307922363 disk.rbd0.in‚0 disk.sdb.in‚ 0.26953125 disk.rbd2.in‚0 disk.rbd2.out‚0.640625% proc.memory.availableMB‚ 5.6953125# proc.memory.rssMB‚ 2992.33203125 disk.sdb2.in‚ 0.26953125 disk.sda.in‚1.671875 disk.sda1.out‚0 disk.sda2.out‚ 252.556640625 network.recv‚804473$ disk./.usageGB‚2994.634204864502 disk.sda.out‚ 252.556640625 cpu‚25.14101372582971À£Î“’‚(:ˆ Û¢éÔâúm gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)ÆÄW“‚(:Ì Û¢éÔ´Êìm disk.dm-0.in‚ 65.56640625 disk.sdb.in‚ 0.26953125 disk.sda.in‚1.671875) disk./.usagePercent‚83.73663202218812% disk./.usageGB‚2994.8275604248047 disk.sdb.out‚ 254.15234375 disk.nvme1n1.in‚ 9.0078125 disk.md0.in‚ 1.94140625 disk.sdb2.in‚ 0.26953125 disk.sdb2.out‚ 254.15234375# memory_percent‚99.9638319015503 disk.md0p1.out‚ 244.265625 disk.rbd2.out‚ 0.7197265625 disk.sdb1.in‚0& proc.memory.availableMB‚ 5.92578125# proc.memory.rssMB‚ 2992.33203125* proc.memory.percent‚18.263745307922363 disk.dm-0.out‚26589.65234375 disk.sda2.in‚1.671875 disk.sda.out‚ 254.15234375 disk.sdb1.out‚0# disk.nvme0n1.out‚10904.04296875! disk.nvme1n1.out‚ 15685.609375 disk.md0.out‚ 244.265625 disk.sda1.in‚0 cpu‚25.14029724600175 disk.rbd2.in‚0 disk.sda1.out‚0 disk.md0p1.in‚ 1.94140625 disk.nvme0n1.in‚ 56.55859375 network.sent‚ 3.966311e+06 disk.sda2.out‚ 254.15234375 disk.rbd0.out‚ 0.357421875 proc.cpu.threads‚17 disk.rbd0.in‚0 network.recv‚808713ÃË2”- _wandbruntime‚2330  _runtime‚2330 )Æ•À & lr-AdamW/pg1‚0.0010369547302872418 & lr-AdamW/pg2‚0.0010369547302872418 trainer/global_step‚2049 % _timestamp‚1.7884982724962463e+09 _runtime‚ 2330.27371392 _step‚163K}Ë–Å & lr-AdamW/pg2‚0.0010369547302872418 trainer/global_step‚2049 % _timestamp‚1.7884982724962463e+09 _runtime‚ 2330.27371392 _step‚163 & lr-AdamW/pg1‚0.0010369547302872418£ fZ2—- _wandbruntime‚2330  _runtime‚2330sôŒV ˜èO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2321374741305 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37665714 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42313782 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2330095126016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12323008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51587697 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚173092424979456 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚190967644915200 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2439316475904 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚165184748057616 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2322282716301 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _step‚164 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25342706 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚173091497006080 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51614039 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚170761419883008 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42313780 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212005888 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2331005096448 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚190967880861184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51614039 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42313782 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚627 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884982724979203e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2331005096448 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170761419883008 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚625èŸPGó'™íO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚173092424979456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚188528564385280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2330095126016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12323008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51587697 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚173092424979456 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚190967644915200 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2439316475904 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚165184748057616 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2322282716301 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _step‚164 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25342706 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚173091497006080 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51614039 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚170761419883008 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42313780 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212005888 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2331005096448 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚190967880861184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51614039 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42313782 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚627 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884982724979203e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2331005096448 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170761419883008 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚625 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2321374741305 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37665714 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42313782 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170843020965888 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚167586257666743 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42333940 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2332126569984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42333940 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚2331.443759321 _step‚165 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93946468 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚173175165538816 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚649 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 trainer/global_step‚2049 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93946468 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚191059287105536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37683616 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256904192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2440497015808 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93972838 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51638896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12328845 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2331216586240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12328198 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚165263765652160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93972838 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚173174237552128 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42333942 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚191059030201344 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42333942 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚170843020965888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚170843038968832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚188618790089728 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚188618766149632 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2323399989787 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37682967 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0®„$–(œ÷M G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93946468 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚191059287105536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37683616 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256904192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2440497015808 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93972838 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51638896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12328845 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2331216586240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12328198 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚165263765652160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93972838 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚173174237552128 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42333942 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚191059030201344 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42333942 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚170843020965888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚170843038968832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚188618790089728 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚188618766149632 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2323399989787 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37682967 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51612528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚173175165538816 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170843020965888 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚167586257666743 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42333940 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2332126569984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14Š”cÕ401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42333940 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚2331.443759321 _step‚165 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93946468 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚173175165538816 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚649 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 trainer/global_step‚2049¥¢X÷2- _wandbruntime‚2331  _runtime‚2331¶Æ­æ[žÕ 2loss/SRqqt4/assignment_loss‚1.7979923468083143 &loss/total_loss‚44.640126870178484 0loss/SRqqt2/detection_loss‚2.518768952952491 _step‚166 0loss/SRqqt1/detection_loss‚2.111138198110792 /loss/FBt2/detection_loss‚2.6167458759413824 /loss/FRt3/detection_loss‚1.7300056208784764 0loss/FRt4/assignment_loss‚3.7746170248304094 1loss/FBt4/assignment_loss‚0.48343059709233266 0loss/SRqqt4/detection_loss‚2.741723909974098 /loss/FBt1/detection_loss‚0.6788881607046869 .loss/FBt4/detection_loss‚2.750116071530751 epoch‚1 0loss/FRt3/assignment_loss‚1.8477032248790448 2loss/SRqqt1/assignment_loss‚1.6946358625535611 1loss/FBt1/assignment_loss‚0.19025014362599543 1loss/FBt3/assignment_loss‚0.06473620298008124 /loss/FRt1/detection_loss‚1.7736457117705589 trainer/global_step‚2049 0loss/FRt1/assignment_loss‚2.0301815282839994 0loss/FRt2/assignment_loss‚3.8538709848355026 1loss/SRqqt2/assignment_loss‚2.075346076930011 .loss/FRt2/detection_loss‚2.357784245067682 /loss/FBt3/detection_loss‚2.2280934386783176 $ _timestamp‚1.788498273667895e+09 _runtime‚2331.444946536 1loss/SRqqt3/assignment_loss‚0.995607089771854 0loss/FBt2/assignment_loss‚0.2658541123022549 /loss/FRt4/detection_loss‚3.0145754579986845 1loss/SRqqt3/detection_loss‚1.0444160316772055Ä,¤,`ŸÚ /loss/FBt2/detection_loss‚2.6167458759413824 /loss/FRt3/detection_loss‚1.7300056208784764 0loss/FRt4/assignment_loss‚3.7746170248304094 1loss/FBt4/assignment_loss‚0.48343059709233266 0loss/SRqqt4/detection_loss‚2.741723909974098 /loss/FBt1/detection_loss‚0.6788881607046869 .loss/FBt4/detection_loss‚2.750116071530751 epoch‚1 0loss/FRt3/assignment_loss‚1.8477032248790448 2loss/SRqqt1/assignment_loss‚1.6946358625535611 1loss/FBt1/assignment_loss‚0.19025014362599543 1loss/FBt3/assignment_loss‚0.06473620298008124 /loss/FRt1/detection_loss‚1.7736457117705589 trainer/global_step‚2049 0loss/FRt1/assignment_loss‚2.0301815282839994 0loss/FRt2/assignment_loss‚3.8538709848355026 1loss/SRqqt2/assignment_loss‚2.075346076930011 .loss/FRt2/detection_loss‚2.357784245067682 /loss/FBt3/detection_loss‚2.2280934386783176 $ _timestamp‚1.788498273667895e+09 _runtime‚2331.444946536 1loss/SRqqt3/assignment_loss‚0.995607089771854 0loss/FBt2/assignment_loss‚0.2658541123022549 /loss/FRt4/detection_loss‚3.0145754579986845 1loss/SRqqt3/detection_loss‚1.0444160316772055 2loss/SRqqt4/assignment_loss‚1.7979923468083143 &loss/total_loss‚44.640126870178484 0loss/SRqqt2/detection_loss‚2.518768952952491 _step‚166 0loss/SRqqt1/detection_loss‚2.111138198110792¦éøÒ‰“ ‚(:ˆ ê¢éÔ‚Æn gpu.0.gpu‚23.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.813' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.70325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0] va¡‚(:Ö ê¢éÔŸîçn disk.rbd2.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0$ memory_percent‚99.96576309204102 disk.sda2.out‚ 256.076171875 disk.sda2.in‚1.671875 disk.dm-0.in‚ 65.56640625 proc.cpu.threads‚17 disk.md0p1.out‚ 246.12890625 disk.sdb.out‚ 256.076171875 disk.md0.in‚ 1.94140625 disk.rbd2.out‚ 0.7197265625 disk.sdb2.in‚ 0.26953125 disk.sda.in‚1.671875 disk.rbd0.out‚ 0.44482421875 disk.sdb1.out‚0% disk./.usageGB‚2994.9117279052734 disk.sda.out‚ 256.076171875 disk.sdb1.in‚0$ proc.memory.availableMB‚5.609375# proc.memory.rssMB‚ 2992.33203125* proc.memory.percent‚18.263745307922363 cpu‚25.13938563823722 disk.nvme0n1.in‚ 56.55859375 disk.sdb2.out‚ 256.076171875 disk.sda1.out‚0 disk.dm-0.out‚26782.22265625 network.recv‚813496) disk./.usagePercent‚83.73898538016886# disk.nvme0n1.out‚10990.52734375 disk.md0p1.in‚ 1.94140625 disk.nvme1n1.in‚ 9.0078125 network.sent‚ 3.983224e+06 disk.sdb.in‚ 0.26953125" disk.nvme1n1.out‚ 15791.6953125 disk.md0.out‚ 246.12890625¸MÝé“¢‚(:ˆ ù¢éÔñâÓm gpu.0.gpu‚18.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.383' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.09575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê`£‚(:Õ ù¢éÔïÌn disk.rbd0.out‚ 0.44482421875 cpu‚25.138522833249734 disk.sdb2.out‚ 257.509765625 disk.nvme1n1.in‚ 9.0078125 disk.md0.out‚ 247.51953125# memory_percent‚99.9650239944458 network.recv‚818134 disk.sdb1.in‚0 disk.dm-0.in‚ 65.56640625 disk.md0.in‚ 1.94140625 proc.cpu.threads‚17" disk.nvme1n1.out‚ 15896.6328125) disk./.usagePercent‚83.75478564475523$ disk./.usageGB‚2995.476821899414 disk.sda1.out‚0 disk.dm-0.out‚ 26922.484375 network.sent‚ 4.076949e+06 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sdb.in‚ 0.26953125 disk.nvme0n1.in‚ 56.55859375 disk.rbd2.in‚0& proc.memory.availableMB‚ 5.73046875 disk.sda.out‚ 257.509765625 disk.sda1.in‚0 disk.sda2.in‚1.671875" disk.nvme0n1.out‚ 11025.8515625 disk.rbd2.out‚ 0.78271484375# proc.memory.rssMB‚ 2992.33203125 disk.sda2.out‚ 257.509765625* proc.memory.percent‚18.263745307922363 disk.sdb2.in‚ 0.26953125 disk.md0p1.in‚ 1.94140625 disk.md0p1.out‚ 247.51953125 disk.sdb.out‚ 257.509765625 disk.sda.in‚1.6718751ë ,’¤‚(:‡ ˆ£éÔÁÞ‰m gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†ÈZTd¥‚(:Ù ˆ£éÔ؈ám disk.rbd2.out‚ 0.78271484375 network.sent‚ 4.085176e+06 disk.md0.in‚ 1.94140625 cpu‚25.137767394369668 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.dm-0.out‚27230.11328125 disk.sda2.in‚1.671875 disk.md0.out‚ 249.2265625! disk.sdb2.out‚259.25927734375 disk.sda.out‚259.25927734375 disk.md0p1.out‚ 249.2265625 disk.dm-0.in‚ 65.56640625 disk.nvme0n1.in‚ 56.55859375# memory_percent‚99.9650239944458& proc.memory.availableMB‚ 5.73046875 disk.rbd0.out‚ 0.5078125 disk.sda.in‚1.671875$ disk./.usageGB‚2995.664321899414 disk.sdb1.out‚0 disk.md0p1.in‚ 1.94140625 disk.sdb.out‚259.25927734375! disk.sda2.out‚259.25927734375# disk.nvme1n1.out‚16106.49609375 disk.sdb2.in‚ 0.26953125# proc.memory.rssMB‚ 2992.33203125" disk.nvme0n1.out‚ 11123.6171875 disk.sda1.in‚0* proc.memory.percent‚18.263745307922363 disk.rbd0.in‚0 disk.sdb.in‚ 0.26953125 disk.nvme1n1.in‚ 9.0078125 disk.sda1.out‚0 network.recv‚822484( disk./.usagePercent‚83.7600282232301 disk.rbd2.in‚0ÅàŽö’¦‚(:‡ —£éÔнƒl gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.434' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.8585 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0LÉ>ØZ§‚(:Ï —£éÔÜö±m) disk./.usagePercent‚83.76704701954635% disk./.usageGB‚2995.9153480529785 disk.dm-0.out‚27431.11328125" disk.nvme0n1.out‚ 11219.6953125 disk.sdb2.out‚ 261.21484375 disk.sda.out‚ 261.21484375 disk.sda.in‚1.671875& proc.memory.availableMB‚ 5.48046875 disk.sdb.in‚ 0.26953125 disk.sda2.in‚1.671875 disk.md0.out‚ 251.12109375$ memory_percent‚99.96654987335205 network.recv‚826664 disk.md0p1.in‚ 1.94140625 disk.nvme0n1.in‚ 56.55859375* proc.memory.percent‚18.263745307922363 cpu‚25.13702184593456 disk.dm-0.in‚ 65.56640625 disk.nvme1n1.in‚ 9.0078125 disk.rbd2.in‚0 disk.rbd2.out‚ 0.845703125 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.sdb.out‚ 261.21484375 disk.sda2.out‚ 261.21484375 disk.md0.in‚ 1.94140625 network.sent‚ 4.091135e+06 disk.md0p1.out‚ 251.12109375# disk.nvme1n1.out‚16211.41796875 disk.sdb2.in‚ 0.26953125 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.rbd0.out‚ 0.5078125# proc.memory.rssMB‚ 2992.33203125ö¤~f2¨- _wandbruntime‚2388  _runtime‚23887ö­Å©¿ _step‚167 % lr-AdamW/pg1‚0.001035077350626465 % lr-AdamW/pg2‚0.001035077350626465 trainer/global_step‚2099 % _timestamp‚1.7884983306753564e+09 _runtime‚2388.452748981FË\)滀 % _timestamp‚1.7884983306753564e+09 _runtime‚2388.452748981 _step‚167 % lr-AdamW/pg1‚0.001035077350626465 % lr-AdamW/pg2‚0.001035077350626465 trainer/global_step‚2099§‚Óç,2«- _wandbruntime‚2388  _runtime‚23886v4í'¬çO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚174842356171264 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38560332 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195537672630272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96154379 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚177228538722304 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43321780 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚169135627784816 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12613759 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195537921159168 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚171512879781229 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43321780 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2498071551488 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚177228538722304 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚171513804795585 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚177229466695680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2386182551040 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚2388.454483569 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174842374174208 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2387092521472 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96180723 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2387092521472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248528896 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 $ _timestamp‚1.788498330676841e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43321782 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2377251996413 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0BG€­ìO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚2388.454483569 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174842374174208 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2387092521472 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96180723 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2387092521472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248528896 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 $ _timestamp‚1.788498330676841e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43321782 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2377251996413 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚174842356171264 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38560332 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195537672630272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96154379 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚177228538722304 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43321780 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚169135627784816 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12613759 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195537921159168 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚171512879781229 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43321780 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2498071551488 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚177228538722304 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚171513804795585 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚177229466695680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2386182551040 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _runtime‚2389.617939264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚169214645379360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 _step‚169 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52857430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25958021 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚171593014649051 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52857430 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚629 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚627 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚177311279268352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38578166 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43341942 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38577537 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚195629313906176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2387304011264 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 265292800 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2379277244895 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43341942 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2099 $ _timestamp‚1.788498331838856e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚177312207255040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚174923975257088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2388213995008 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96225740 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 241352704 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚174923993260032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43341940 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96199370 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12619518 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚193130075312128 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43341940 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43341940 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _runtime‚2389.617939264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚169214645379360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 _step‚169 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52857430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25958021 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚171593014649051 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52857430 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚629 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚627 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚177311279268352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38578166 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43341942 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38577537 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚195629313906176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2387304011264 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 265292800 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2379277244895 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43341942 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2099 $ _timestamp‚1.788498331838856e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚177312207255040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚174923975257088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2388213995008 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96225740 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 241352704 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚174923993260032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43341940 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96199370©V1Â2±- _wandbruntime‚2389  _runtime‚2389¼’:ZW²Ñ 0loss/SRqqt2/detection_loss‚2.774092553981713 /loss/FRt3/detection_loss‚1.8189415258044999 /loss/FRt4/detection_loss‚3.3568011793223294 0loss/FRt2/assignment_loss‚3.2939389997058446 0loss/FRt3/assignment_loss‚1.9481953094402948 1loss/SRqqt1/assignment_loss‚2.241699476622873 /loss/FBt1/detection_loss‚0.7515657196898956 /loss/FBt2/detection_loss‚2.8858058886094526 .loss/FBt3/detection_loss‚3.799700689315796 /loss/FBt4/detection_loss‚2.8368593975901604 %loss/total_loss‚47.78461527925115 $ _timestamp‚1.788498331842219e+09 epoch‚1 trainer/global_step‚2099 2loss/SRqqt2/assignment_loss‚2.2583026760923013 1loss/FBt1/assignment_loss‚0.20421891497004152 1loss/FBt4/assignment_loss‚0.05130970963273285 0loss/FRt1/assignment_loss‚3.2278761518628976 0loss/FBt3/assignment_loss‚0.1944787195592653 0loss/SRqqt1/detection_loss‚1.971899343861474 _runtime‚ 2389.61930778 /loss/FRt4/assignment_loss‚3.275012520226565 1loss/SRqqt3/assignment_loss‚1.225295653826267 2loss/SRqqt4/assignment_loss‚1.8374798049529393 _step‚170 1loss/SRqqt3/detection_loss‚0.9070303967588993 0loss/SRqqt4/detection_loss‚2.683400683104992 .loss/FRt2/detection_loss‚2.334185262603892 0loss/FBt2/assignment_loss‚0.1108622925373559 /loss/FRt1/detection_loss‚1.7956624091786675.W¥3\³Ö 1loss/FBt4/assignment_loss‚0.05130970963273285 0loss/FRt1/assignment_loss‚3.2278761518628976 0loss/FBt3/assignment_loss‚0.1944787195592653 0loss/SRqqt1/detection_loss‚1.971899343861474 _runtime‚ 2389.61930778 /loss/FRt4/assignment_loss‚3.275012520226565 1loss/SRqqt3/assignment_loss‚1.225295653826267 2loss/SRqqt4/assignment_loss‚1.8374798049529393 _step‚170 1loss/SRqqt3/detection_loss‚0.9070303967588993 0loss/SRqqt4/detection_loss‚2.683400683104992 .loss/FRt2/detection_loss‚2.334185262603892 0loss/FBt2/assignment_loss‚0.1108622925373559 /loss/FRt1/detection_loss‚1.7956624091786675 0loss/SRqqt2/detection_loss‚2.774092553981713 /loss/FRt3/detection_loss‚1.8189415258044999 /loss/FRt4/detection_loss‚3.3568011793223294 0loss/FRt2/assignment_loss‚3.2939389997058446 0loss/FRt3/assignment_loss‚1.9481953094402948 1loss/SRqqt1/assignment_loss‚2.241699476622873 /loss/FBt1/detection_loss‚0.7515657196898956 /loss/FBt2/detection_loss‚2.8858058886094526 .loss/FBt3/detection_loss‚3.799700689315796 /loss/FBt4/detection_loss‚2.8368593975901604 %loss/total_loss‚47.78461527925115 $ _timestamp‚1.788498331842219e+09 epoch‚1 trainer/global_step‚2099 2loss/SRqqt2/assignment_loss‚2.2583026760923013 1loss/FBt1/assignment_loss‚0.20421891497004152ª­w™’´‚(:‡ ¦£éÔÑÈân gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.774' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.9435 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì€`µ‚(:Õ ¦£éÔ—Îþn) disk./.usagePercent‚83.76701854118429 disk.sdb2.in‚ 0.26953125 disk.sdb1.out‚0 disk.dm-0.out‚27784.04296875! disk.nvme1n1.out‚ 16368.953125 proc.cpu.threads‚17 network.recv‚831405 disk.sda.out‚ 262.716796875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 65.56640625 disk.rbd2.in‚0* proc.memory.percent‚18.263745307922363% proc.memory.availableMB‚ 5.7109375 disk.sdb2.out‚ 262.716796875 disk.sda2.in‚1.671875# disk.nvme0n1.out‚11415.08984375 disk.rbd2.out‚ 0.845703125 disk.md0.out‚ 252.54296875% disk./.usageGB‚2995.9143295288086 disk.sdb.in‚ 0.26953125# proc.memory.rssMB‚ 2992.33203125 disk.sda.in‚1.671875 disk.sdb1.in‚0 disk.sdb.out‚ 262.716796875 disk.nvme1n1.in‚ 9.0078125 disk.sda1.in‚0 disk.md0p1.in‚ 1.94140625 disk.md0p1.out‚ 252.54296875 disk.nvme0n1.in‚ 56.55859375 cpu‚25.136242797744185 disk.rbd0.out‚ 0.57080078125 disk.sda2.out‚ 262.716796875$ memory_percent‚99.96514320373535 disk.md0.in‚ 1.94140625 network.sent‚ 4.09521e+06#ü³¼’¶‚(:‡ µ£éÔ˜–‘m gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.562' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.6405 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«(îäd·‚(:Ù µ£éÔã΃n disk.rbd2.in‚0! disk.sdb2.out‚264.28369140625 network.recv‚836302% disk./.usageGB‚2995.9143295288086 disk.dm-0.out‚ 27930.34375 disk.nvme0n1.in‚ 56.55859375& proc.memory.availableMB‚ 5.69140625 disk.md0p1.in‚ 1.94140625 disk.nvme1n1.in‚ 9.0078125 disk.md0.out‚ 254.0546875 disk.sdb2.in‚ 0.26953125 disk.sda.out‚264.28369140625 cpu‚25.135733306901827# memory_percent‚99.9652624130249 disk.sdb.in‚ 0.26953125 disk.sda1.out‚0" disk.nvme0n1.out‚ 11449.2734375 disk.rbd2.out‚ 0.845703125 disk.rbd0.out‚ 0.57080078125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0p1.out‚ 254.0546875 disk.sdb.out‚264.28369140625! disk.sda2.out‚264.28369140625 proc.cpu.threads‚17 disk.dm-0.in‚ 65.56640625 disk.sda2.in‚1.671875 disk.sda.in‚1.671875 disk.sda1.in‚0# proc.memory.rssMB‚ 2992.33203125* proc.memory.percent‚18.263745307922363 network.sent‚ 4.201808e+06) disk./.usagePercent‚83.76701854118429 disk.rbd0.in‚0" disk.nvme1n1.out‚ 16481.0703125 disk.md0.in‚ 1.94140625ûp`z“¸‚(:ˆ Ä£éÔƒ“ˆm gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚168.715' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.17875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š®—!^¹‚(:Ó Ä£éÔÇún disk.md0p1.out‚ 255.921875 disk.dm-0.out‚ 28106.3671875 disk.sda2.in‚1.671875) disk./.usagePercent‚83.72274247437417 disk.rbd2.out‚ 0.845703125 disk.rbd0.in‚0 disk.sdb.in‚ 0.26953125 disk.md0.in‚ 1.94140625 disk.sda1.out‚0" disk.nvme0n1.out‚ 11579.6796875 network.recv‚840761 disk.nvme1n1.in‚ 9.0078125 disk.nvme1n1.out‚ 16526.6875! disk.sdb2.out‚266.21923828125 disk.rbd0.out‚ 0.57080078125 disk.nvme0n1.in‚ 56.55859375# proc.memory.rssMB‚ 2992.33203125* proc.memory.percent‚18.263745307922363 disk.sdb1.in‚0 disk.sdb.out‚266.21923828125 disk.sda.in‚1.671875 disk.sda.out‚266.21923828125 network.sent‚ 4.209924e+06% disk./.usageGB‚2994.3308029174805 disk.md0.out‚ 255.921875 disk.rbd2.in‚0 disk.md0p1.in‚ 1.94140625 disk.dm-0.in‚ 65.56640625! disk.sda2.out‚266.21923828125 cpu‚25.135088328203967 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0$ memory_percent‚99.96528625488281" proc.memory.availableMB‚5.6875 disk.sdb2.in‚ 0.26953125ˆe2º- _wandbruntime‚2443  _runtime‚2443šÂ÷ÐÅ»¿ % _timestamp‚1.7884983855962036e+09 _runtime‚2443.373944701 _step‚171 % lr-AdamW/pg1‚0.001033093916867802 % lr-AdamW/pg2‚0.001033093916867802 trainer/global_step‚21496Yë(Ê¼Ä _step‚171 % lr-AdamW/pg1‚0.001033093916867802 % lr-AdamW/pg2‚0.001033093916867802 trainer/global_step‚2149 % _timestamp‚1.7884983855962036e+09 _runtime‚2443.373944701«ãÉâ2½- _wandbruntime‚2443  _runtime‚2443Eß$¦Ñ ¾èO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2443179946496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2442269976064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12905642 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚629 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200107627926016 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2556517036032 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚178923310462464 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44329782 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39454848 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238043136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚2443.375683739 _step‚172 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚181365580438528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚178923310462464 JBDeviceStatsMonitor.on_traiv>cìn_batch_start/inactive_split.all.current‚631 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181365580438528 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39454217 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2442269976064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚175519636763537 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54103843 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚181366508411904 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚175520561777893 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2556731139072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚200107865969152 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98407281 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44329780 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚178923328465408 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214103040 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26549206 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚173086507512016 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2443179946496 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚181366508411904 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2433129251521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26549204 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197551134830080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 trainer/global_step‚2149 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197551110889984 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570ŸÀ¸ó'¿íO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚181366508411904 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2433129251521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26549204 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197551134830080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 trainer/global_step‚2149 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197551110889984 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2443179946496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2442269976064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12905642 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚629 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200107627926016 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2556517036032 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚178923310462464 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44329782 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39454848 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238043136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚2443.375683739 _step‚172 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚181365580438528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚178923310462464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚631 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181365580438528 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39454217 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2442269976064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚175519636763537 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54103843 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚181366508411904 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚175520561777893 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2556731139072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚200107865969152 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98407281 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44329780 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚178923328465408 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214103040 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26549206 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚173086507512016 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2443179946496 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚2444.338014984 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚173165525106560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 211992576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚179004929548288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚611 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54102332 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98478642 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54128700 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2557712002048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚175599771631359 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚179004947551232 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2444301420032 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98452272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚179004929548288 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚197641336594432 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2443391436288 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98452272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚175600696645923 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181449248971264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚200199048596480 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚200199284529152 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚173 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12911298 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 235932672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2443391436288 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44349940 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44349942 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44349942 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98478642 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26561271 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚197641360534528 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2557923994624 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚173165542145920 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2435154500003 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39471958™ñ=o£Â÷M FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181449248971264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚200199048596480 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚200199284529152 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚173 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12911298 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 235932672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2443391436288 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44349940 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44349942 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44349942 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98478642 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26561271 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚197641360534528 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2557923994624 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚173165542145920 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2435154500003 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39471958 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 N>DeviceStatsMonitor.on_train_batch_endòØè‘Z/allocated_bytes.all.peak‚ 14135737344 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚2444.338014984 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚173165525106560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 211992576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚179004929548288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚611 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54102332 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98478642 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54128700 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2557712002048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚175599771631359 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚179004947551232 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2444301420032 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98452272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚179004929548288 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚197641336594432 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2443391436288 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98452272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚175600696645923­Ô`ÊI2Ã- _wandbruntime‚2444  _runtime‚2444!ºCZÄÔ /loss/FRt1/detection_loss‚2.0369699587249577 .loss/FRt4/detection_loss‚2.987963114465986 .loss/FBt2/detection_loss‚2.336774961153666 1loss/SRqqt2/detection_loss‚2.3707797509782456 %loss/total_loss‚46.80628216782293 0loss/FRt1/assignment_loss‚2.5200220468092938 /loss/FRt2/assignment_loss‚3.863643990244184 2loss/SRqqt4/assignment_loss‚2.5437024449929595 1loss/FBt2/assignment_loss‚0.36420736153037675 /loss/FRt3/detection_loss‚1.7618126972620167 .loss/FBt3/detection_loss‚3.245736062526703 epoch‚1 /loss/FRt4/assignment_loss‚4.582903231893267 1loss/SRqqt4/detection_loss‚2.2257554829120636 /loss/FBt1/detection_loss‚1.0975169327463592 2loss/SRqqt3/assignment_loss‚1.2375914188085404 3loss/FBt3/assignment_loss‚0.0005492378495546291 0loss/FBt4/assignment_loss‚0.2316980109526412 -loss/FRt2/detection_loss‚2.67549130831446 0loss/SRqqt3/detection_loss‚1.103854008678314 _step‚174 2loss/SRqqt1/assignment_loss‚1.1862107058686595 2loss/SRqqt2/assignment_loss‚1.7431565027903109 1loss/FBt1/assignment_loss‚0.23538666194878255 /loss/FBt4/detection_loss‚2.2966428138315678 trainer/global_step‚2149 _runtime‚2444.339015868 0loss/FRt3/assignment_loss‚2.0426861910677667 0loss/SRqqt1/detection_loss‚2.115227271472254 $ _timestamp‚1.788498386561773e+09b¡Ö`_ÅÙ 3loss/FBt3/assignment_loss‚0.0005492378495546291 0loss/FBt4/assignment_loss‚0.2316980109526412 -loss/FRt2/detection_loss‚2.67549130831446 0loss/SRqqt3/detection_loss‚1.103854008678314 _step‚174 2loss/SRqqt1/assignment_loss‚1.1862107058686595 2loss/SRqqt2/assignment_loss‚1.7431565027903109 1loss/FBt1/assignment_loss‚0.23538666194878255 /loss/FBt4/detection_loss‚2.2966428138315678 trainer/global_step‚2149 _runtime‚2444.339015868 0loss/FRt3/assignment_loss‚2.0426861910677667 0loss/SRqqt1/detection_loss‚2.115227271472254 $ _timestamp‚1.788498386561773e+09 /loss/FRt1/detection_loss‚2.0369699587249577 .loss/FRt4/detection_loss‚2.987963114465986 .loss/FBt2/detection_loss‚2.336774961153666 1loss/SRqqt2/detection_loss‚2.3707797509782456 %loss/total_loss‚46.80628216782293 0loss/FRt1/assignment_loss‚2.5200220468092938 /loss/FRt2/assignment_loss‚3.863643990244184 2loss/SRqqt4/assignment_loss‚2.5437024449929595 1loss/FBt2/assignment_loss‚0.36420736153037675 /loss/FRt3/detection_loss‚1.7618126972620167 .loss/FBt3/detection_loss‚3.245736062526703 epoch‚1 /loss/FRt4/assignment_loss‚4.582903231893267 1loss/SRqqt4/detection_loss‚2.2257554829120636 /loss/FBt1/detection_loss‚1.0975169327463592 2loss/SRqqt3/assignment_loss‚1.2375914188085404®EQgÆ‚(:Ü Ó£éÔÀÛám disk.md0.out‚ 257.30859375 disk.dm-0.in‚ 65.56640625 disk.sdb.in‚ 0.26953125 disk.sdb1.out‚0# disk.nvme0n1.out‚11634.55078125) disk./.usagePercent‚83.72273180832096 disk.md0.in‚ 1.94140625! disk.sdb2.out‚267.66943359375 disk.sda1.in‚0 disk.sda.in‚1.671875 disk.sda.out‚267.66943359375 disk.sda1.out‚0 disk.rbd0.in‚0* proc.memory.percent‚18.263745307922363 disk.md0p1.in‚ 1.94140625 network.sent‚ 4.213591e+06 disk.sdb2.in‚ 0.26953125! disk.sda2.out‚267.66943359375 disk.nvme0n1.in‚ 56.55859375 disk.nvme1n1.in‚ 9.0078125# disk.nvme1n1.out‚16614.98046875# proc.memory.rssMB‚ 2992.33203125 cpu‚25.134317095445056 network.recv‚845088$ disk./.usageGB‚2994.330421447754 disk.rbd0.out‚ 0.57080078125 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 0.845703125 disk.md0p1.out‚ 257.30859375 disk.dm-0.out‚ 28249.53125 disk.sdb.out‚267.66943359375# memory_percent‚99.9699354171753& proc.memory.availableMB‚ 4.92578125 disk.sda2.in‚1.671875 proc.cpu.threads‚17k7ù‘Ç‚(:† Ó£éԥȿn gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚175.512' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A]€piÈ‚(:Þ â£éÔ°ã·m" disk.nvme1n1.out‚ 16615.3046875 disk.sda.in‚1.671875 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.md0p1.out‚ 258.60546875 disk.dm-0.out‚28250.66015625 disk.sdb.in‚ 0.26953125 network.recv‚850361 disk.md0p1.in‚ 1.94140625 disk.sda2.in‚1.671875 disk.rbd2.in‚0 disk.rbd2.out‚ 0.845703125) disk./.usagePercent‚83.72399338909489 disk.sdb.out‚269.02001953125 proc.cpu.threads‚17 network.sent‚ 4.322496e+06 disk.sda.out‚269.02001953125 disk.nvme1n1.in‚ 9.0078125 disk.md0.in‚ 1.94140625% disk./.usageGB‚2994.3755416870117$ memory_percent‚99.96805191040039! disk.sdb2.out‚269.02001953125! disk.sda2.out‚269.02001953125# disk.nvme0n1.out‚11635.35546875 cpu‚25.133690852775647 disk.sda1.out‚0 disk.rbd0.out‚ 0.57080078125 disk.dm-0.in‚ 65.56640625 disk.md0.out‚ 258.60546875* proc.memory.percent‚18.263745307922363 disk.sdb2.in‚ 0.26953125 disk.sdb1.out‚0 disk.nvme0n1.in‚ 56.55859375$ proc.memory.availableMB‚5.234375# proc.memory.rssMB‚ 2992.33203125®;¢k“É‚(:ˆ â£éÔÞšŒt gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚125.271' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.31775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L|wÃ’Ê‚(:‡ ñ£éÔ³—ºk gpu.0.gpu‚28.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚181.414' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.3535 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tÏÄgË‚(:Ü ñ£éÔ®×õl disk.dm-0.in‚ 65.56640625 network.sent‚ 4.325651e+06 network.recv‚854037 disk.md0.out‚ 260.0546875 disk.sdb2.in‚ 0.26953125! disk.sdb2.out‚270.52392578125# disk.nvme1n1.out‚16616.42578125 disk.md0.in‚ 1.94140625 disk.md0p1.in‚ 1.94140625% proc.memory.availableMB‚ 5.1015625$ memory_percent‚99.96886253356934 cpu‚25.132999509595383 proc.cpu.threads‚17# disk.nvme0n1.out‚11721.71484375 disk.nvme1n1.in‚ 9.0078125 disk.rbd2.in‚0 disk.rbd0.out‚ 0.57080078125 disk.md0p1.out‚ 260.0546875 disk.dm-0.out‚ 28338.140625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0% disk./.usageGB‚2994.2699699401855 disk.nvme0n1.in‚ 56.55859375 disk.sda.out‚270.52392578125# proc.memory.rssMB‚ 2992.33203125* proc.memory.percent‚18.263745307922363 disk.rbd2.out‚ 0.845703125 disk.sdb.in‚ 0.26953125 disk.sdb.out‚270.52392578125) disk./.usagePercent‚83.72104155886842 disk.sda.in‚1.671875 disk.sda2.in‚1.671875! disk.sda2.out‚270.52392578125 disk.sdb1.in‚0 disk.sdb1.out‚0èš•æ“Ì‚(:ˆ €¤éÔ¸À´k gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚117.525' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.38125 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_ðc4YÍ‚(:Î €¤éÔŸ‹ël* proc.memory.percent‚18.263745307922363 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.md0p1.out‚ 261.87890625 disk.sda2.in‚1.671875# proc.memory.rssMB‚ 2992.33203125 proc.cpu.threads‚17 disk.rbd0.out‚ 0.57080078125 disk.sdb2.in‚ 0.26953125$ memory_percent‚99.96886253356934 network.sent‚ 4.333994e+06 disk.sda.out‚ 272.4140625# disk.nvme0n1.out‚11829.13671875 disk.rbd2.out‚ 0.845703125 network.recv‚858479$ disk./.usageGB‚2994.535991668701 disk.sda1.in‚0 disk.sdb1.in‚0 disk.md0p1.in‚ 1.94140625 disk.md0.out‚ 261.87890625 disk.sdb.out‚ 272.4140625 disk.sda2.out‚ 272.4140625 disk.nvme0n1.in‚ 56.55859375 disk.md0.in‚ 1.94140625 cpu‚25.132406909434827 disk.dm-0.out‚ 28445.7421875% proc.memory.availableMB‚ 5.1015625 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sdb.in‚ 0.26953125# disk.nvme1n1.out‚16616.60546875 disk.sdb2.out‚ 272.4140625) disk./.usagePercent‚83.72847963773647 disk.sda.in‚1.671875 disk.dm-0.in‚ 65.56640625 disk.nvme1n1.in‚ 9.00781258xY2Î- _wandbruntime‚2497  _runtime‚2497Œ}8‚ÇÏÁ & lr-AdamW/pg1‚0.0010310048423558757 & lr-AdamW/pg2‚0.0010310048423558757 trainer/global_step‚2199 % _timestamp‚1.7884984397302978e+09 _runtime‚2497.507574052 _step‚175³ ÌÐÆ _step‚175 & lr-AdamW/pg1‚0.0010310048423558757 & lr-AdamW/pg2‚0.0010310048423558757 trainer/global_step‚2199 % _timestamp‚1.7884984397302978e+09 _runtime‚2497.507574052¯ k?2Ñ- _wandbruntime‚2497  _runtime‚2497·Õ¢ò'ÒìO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185502622154752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚606 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2489914481625 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 % _timestamp‚1.7884984397316399e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27152454 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100686527 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚179526393745845 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚177037387239216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2498357401088 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214103040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238043136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55348745 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55322403 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚185503550128128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚202062420052480 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚2497.509076169 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 trainer/global_step‚2199 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚177037404278576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚183004282756608 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2498357401088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2499267371520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13197348 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚185503550128128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚185502622154752 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚183004264753664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13196742 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 _step‚176 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40349804 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2615645393920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2489006506629 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55322403 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚183004282756608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45337782 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100660183 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚183004264753664 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2499267371520 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45337780 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚204678065446400 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚183004282756608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45337782 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100660183 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚183004264753664 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2499267371520 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45337780 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚204678065446400 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185502622154752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚606 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2489914481625 LDDeviceStatsMonitoržÐK| .on_train_batch_start/segment.small_pool.allocated‚570 % _timestamp‚1.7884984397316399e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27152454 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100686527 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚179526393745845 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚177037387239216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2498357401088 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214103040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238043136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55348745 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55322403 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚185503550128128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615°Œüà 2Ô- _wandbruntime‚2498  _runtime‚2498iòö±û&ÕõM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚202152621816832 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100731544 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100705174 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185586290687488 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45357942 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13202555 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2500388845056 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45357940 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2499478861312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2491031755111 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55347234 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45357942 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2616826894336 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27164521 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226672640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 $ _timestamp‚1.788498440684292e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚177116421873120 _step‚177 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2490123779907 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27164519 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40367074 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100705174 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40367698 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2500388845056 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2499478861312 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚185585362700800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚183085883839488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13203177 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚179606528613667 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2616600221696 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2199 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100731544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204769222038528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚179607453628231 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 _runtime‚2498.462282441 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55373602 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚183085883839488 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55347234 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚622 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45357940 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55373602 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204769472651264+$vi'ÖúM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2500388845056 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45357940 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2499478861312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2491031755111 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55347234 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45357942 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2616826894336 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27164521 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226672640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 $ _timestamp‚1.788498440684292e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚177116421873120 _step‚177 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2490123779907 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27164519 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40367074 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100705174 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40367698 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2500388845056 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2499478861312 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚185585362700800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚183085883839488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13203177 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚179606528613667 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2616600221696 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2199 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100731544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204769222038528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚179607453628231 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 _runtime‚2498.462282441 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55373602 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚183085883839488 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55347234 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚622 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45357940 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55373602 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204769472651264 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚202152621816832 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100731544 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100705174 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185586290687488 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45357942 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13202555 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152±ø‹yF2×- _wandbruntime‚2498  _runtime‚2498f]qTZØÔ 1loss/FBt2/assignment_loss‚0.12248975941975004 0loss/SRqqt2/detection_loss‚2.694844176769257 0loss/FRt3/assignment_loss‚1.6284473644148918 /loss/FBt1/detection_loss‚1.0253850503135113 _runtime‚2498.463891089 .loss/FBt3/detection_loss‚2.967126250267029 trainer/global_step‚2199 .loss/FRt1/detection_loss‚1.748212996503695 2loss/SRqqt1/assignment_loss‚1.6574728692066856 0loss/FRt1/assignment_loss‚2.2760813781555664 2loss/FBt3/assignment_loss‚0.013631033555157046 0loss/SRqqt1/detection_loss‚2.243133870884776 epoch‚1 0loss/FRt4/assignment_loss‚3.6648553609848022 .loss/FRt4/detection_loss‚3.001490214041301 /loss/FBt4/detection_loss‚3.2720184127489724 &loss/total_loss‚46.759950988867246 1loss/FBt1/assignment_loss‚0.13281847675600666 0loss/SRqqt4/detection_loss‚3.045406240483989 .loss/FBt2/detection_loss‚2.657403054691496 _step‚178 0loss/FRt2/assignment_loss‚3.5699767401052074 1loss/SRqqt2/assignment_loss‚2.308750352859497 2loss/SRqqt3/assignment_loss‚1.2603604455288608 2loss/SRqqt4/assignment_loss‚1.8725836639818938 1loss/FBt4/assignment_loss‚0.17397394858906207 .loss/FRt3/detection_loss‚1.804587532796206 .loss/FRt2/detection_loss‚2.405382501524548 1loss/SRqqt3/detection_loss‚1.2135192942850799 $ _timestamp‚1.788498440686897e+09‘â[{_ÙÙ 0loss/FRt1/assignment_loss‚2.2760813781555664 2loss/FBt3/assignment_loss‚0.013631033555157046 0loss/SRqqt1/detection_loss‚2.243133870884776 epoch‚1 0loss/FRt4/assignment_loss‚3.6648553609848022 .loss/FRt4/detection_loss‚3.001490214041301 /loss/FBt4/detection_loss‚3.2720184127489724 &loss/total_loss‚46.759950988867246 1loss/FBt1/assignment_loss‚0.13281847675600666 0loss/SRqqt4/detection_loss‚3.045406240483989 .loss/FBt2/detection_loss‚2.657403054691496 _step‚178 0loss/FRt2/assignment_loss‚3.5699767401052074 1loss/SRqqt2/assignment_loss‚2.308750352859497 2loss/SRqqt3/assignment_loss‚1.2603604455288608 2loss/SRqqt4/assignment_loss‚1.8725836639818938 1loss/FBt4/assignment_loss‚0.17397394858906207 .loss/FRt3/detection_loss‚1.804587532796206 .loss/FRt2/detection_loss‚2.405382501524548 1loss/SRqqt3/detection_loss‚1.2135192942850799 $ _timestamp‚1.788498440686897e+09 1loss/FBt2/assignment_loss‚0.12248975941975004 0loss/SRqqt2/detection_loss‚2.694844176769257 0loss/FRt3/assignment_loss‚1.6284473644148918 /loss/FBt1/detection_loss‚1.0253850503135113 _runtime‚2498.463891089 .loss/FBt3/detection_loss‚2.967126250267029 trainer/global_step‚2199 .loss/FRt1/detection_loss‚1.748212996503695 2loss/SRqqt1/assignment_loss‚1.6574728692066856²gå ù]Ú‚(:Ò ¤éÔã°Êm) disk./.usagePercent‚83.73207996399823 disk.sda2.in‚1.671875 disk.sdb2.in‚ 0.26953125 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.nvme1n1.in‚ 9.0078125# disk.nvme1n1.out‚16616.72265625! disk.sdb2.out‚273.74951171875 disk.sda.out‚273.74951171875 disk.sdb1.in‚0 disk.md0p1.out‚ 263.15625 cpu‚25.131705193329715 disk.sdb.out‚273.74951171875& proc.memory.availableMB‚ 5.34765625 network.recv‚862842 disk.nvme0n1.in‚ 56.55859375 disk.md0.out‚ 263.15625 disk.dm-0.in‚ 65.56640625 disk.md0.in‚ 1.94140625 disk.sda.in‚1.671875 disk.sda1.out‚0 disk.sdb.in‚ 0.26953125# disk.nvme0n1.out‚11998.77734375 disk.rbd2.in‚0 disk.rbd2.out‚ 0.845703125 disk.rbd0.out‚ 0.57080078125 disk.dm-0.out‚28615.5* proc.memory.percent‚18.263745307922363 proc.cpu.threads‚17 network.sent‚ 4.337595e+06% disk./.usageGB‚2994.6647567749023! disk.sda2.out‚273.74951171875 disk.sda1.in‚0 disk.md0p1.in‚ 1.94140625" memory_percent‚99.967360496521# proc.memory.rssMB‚ 2992.33203125|Q˜c‘Û‚(:† ¤éÔ„Ûým gpu.0.gpu‚39.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.916' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.229 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uëùדÜ‚(:ˆ ž¤éÔšæÕk gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.157' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.78925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CM&TÝ‚(:É ž¤éÔ„éÖm disk.sdb.out‚ 275.234375 disk.nvme0n1.in‚ 56.55859375 disk.md0.in‚ 1.94140625 proc.cpu.threads‚17 disk.md0p1.in‚ 1.94140625 disk.rbd2.out‚ 0.845703125 disk.sda1.in‚0 cpu‚25.131123477277516% disk./.usageGB‚2994.9147033691406 disk.dm-0.in‚ 65.56640625 disk.rbd2.in‚0 disk.sda.in‚1.671875 disk.rbd0.in‚0 disk.sdb1.in‚0) disk./.usagePercent‚83.73906857538391 disk.dm-0.out‚28783.22265625 disk.sdb.in‚ 0.26953125 disk.nvme1n1.in‚ 9.0078125 disk.sdb2.out‚ 275.234375 disk.sda1.out‚0 network.sent‚ 4.447449e+06 disk.sda2.in‚1.671875 disk.nvme0n1.out‚ 12166.15625 disk.rbd0.out‚ 0.57080078125% proc.memory.availableMB‚ 5.3359375 network.recv‚868977 disk.md0p1.out‚ 264.57421875* proc.memory.percent‚18.263745307922363 disk.sda2.out‚ 275.234375 disk.md0.out‚ 264.57421875 disk.sdb2.in‚ 0.26953125 disk.sda.out‚ 275.234375 disk.sdb1.out‚0# proc.memory.rssMB‚ 2992.33203125# disk.nvme1n1.out‚16617.06640625$ memory_percent‚99.96743202209473„9½Þ‚(:„ ­¤éÔÁ äl gpu.0.gpu‚13.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚177.84' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.46 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êl( ]ß‚(:Ò ­¤éÔÝ£™m disk.sdb2.out‚ 276.73828125# proc.memory.rssMB‚ 2992.33203125 disk.rbd0.in‚0 disk.dm-0.in‚ 65.56640625 disk.sda2.out‚ 276.73828125 cpu‚25.130462838303554 network.recv‚872649$ disk./.usageGB‚2995.039909362793 disk.sdb1.out‚0 disk.dm-0.out‚ 28965.9296875 disk.sda2.in‚1.671875 disk.md0.in‚ 1.94140625 disk.sdb2.in‚ 0.26953125 disk.rbd0.out‚ 0.57080078125 disk.sdb.in‚ 0.26953125$ memory_percent‚99.96740818023682( disk./.usagePercent‚83.7425693873692 disk.sda.out‚ 276.73828125 disk.sda1.in‚0 disk.md0p1.in‚ 1.94140625 disk.sdb.out‚ 276.73828125* proc.memory.percent‚18.263745307922363 disk.nvme0n1.in‚ 56.55859375# disk.nvme0n1.out‚12358.76171875 network.sent‚ 4.452722e+06 disk.md0.out‚ 266.01171875 disk.rbd2.out‚ 0.845703125 disk.sdb1.in‚0 disk.md0p1.out‚ 266.01171875# disk.nvme1n1.out‚16617.16796875 disk.rbd2.in‚0 disk.sda.in‚1.671875 disk.sda1.out‚0 disk.nvme1n1.in‚ 9.0078125& proc.memory.availableMB‚ 5.33984375 proc.cpu.threads‚17«šÐUhà‚(:Ý ¼¤éÔ­‡p% disk./.usageGB‚2995.1646423339844 disk.sdb2.in‚ 0.26953125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.nvme1n1.in‚ 9.0078125 disk.sda.in‚1.671875 network.recv‚877279 disk.dm-0.out‚29103.76171875* proc.memory.percent‚18.263745307922363 disk.sda1.in‚0 disk.rbd0.out‚ 0.57080078125# disk.nvme0n1.out‚12486.33984375! disk.nvme1n1.out‚ 16617.421875 disk.md0.in‚ 1.94140625 disk.rbd2.in‚0% proc.memory.availableMB‚ 5.3359375 cpu‚25.12974038048842 disk.md0p1.in‚ 1.94140625 disk.rbd2.out‚ 0.845703125$ memory_percent‚99.96743202209473! disk.sdb2.out‚278.81103515625 disk.sda.out‚278.81103515625 disk.md0p1.out‚ 268.01953125 disk.dm-0.in‚ 65.56640625 disk.sda2.in‚1.671875! disk.sda2.out‚278.81103515625 disk.nvme0n1.in‚ 56.55859375# proc.memory.rssMB‚ 2992.33203125 network.sent‚ 4.456521e+06) disk./.usagePercent‚83.74605697344852 disk.sda1.out‚0 disk.sdb.out‚278.81103515625 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sdb.in‚ 0.26953125 disk.md0.out‚ 268.01953125£ Ó“á‚(:ˆ ¼¤éÔ˜šžp gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚180.473' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.11825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´u÷Ñ2â- _wandbruntime‚2552  _runtime‚2552}iÇãÁ trainer/global_step‚2249 % _timestamp‚1.7884984947926512e+09 _runtime‚2552.569933042 _step‚179 & lr-AdamW/pg1‚0.0010288105624506826 & lr-AdamW/pg2‚0.0010288105624506826ôÍñŠÌäÆ % _timestamp‚1.7884984947926512e+09 _runtime‚2552.569933042 _step‚179 & lr-AdamW/pg1‚0.0010288105624506826 & lr-AdamW/pg2‚0.0010288105624506826 trainer/global_step‚2249³O´Ë2å- _wandbruntime‚2552  _runtime‚2552òžó0-æìO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56593647 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27755706 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚209247803907072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚180988284005776 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviÁ»SÌÅ$ceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2674351355392 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 trainer/global_step‚2249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚206573705274880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2554444826112 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚189639663870976 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚187085237047808 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46345782 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2545791736733 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2544883761737 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2554444826112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚627 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚189640591844352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56567305 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚206573681334784 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚189640591844352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚183533150728153 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚187085219044864 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102913085 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41244852 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚180988266966416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚187085219044864 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚189639663870976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56593647 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46345780 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2674122572288 _runtime‚2552.571236608 _step‚180 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13489146 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚183534075742509 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 % _timestamp‚1.7884984947938235e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚187085237047808 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2555354796544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13488519 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2555354796544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209248056630272 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56567305 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252723200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102939429 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0¶ž©—÷'çñO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚187085237047808 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2555354796544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13488519 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2555354796544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209248056630272 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56567305 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252723200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102939429 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56593647 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27755706 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚209247803907072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚180988284005776 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2674351355392 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 trainer/global_step‚2249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚206573705274880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2554444826112 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚189639663870976 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚187085237047808 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46345782 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2545791736733 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2544883761737 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2554444826112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚627 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚189640591844352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56567305 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚206573681334784 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚189640591844352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚183533150728153 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚187085219044864 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102913085 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41244852 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚180988266966416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚187085219044864 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚189639663870976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56593647 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46345780 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2674122572288 _runtime‚2552.571236608 _step‚180 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13489146 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚183534075742509 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 % _timestamp‚1.7884984947938235e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247´K±§2è- _wandbruntime‚2553  _runtime‚2553”¯Ãgû&éõM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚181067284560960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2555566286336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚209339211009536 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚181067301600320 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2546909010219 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41262763 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218284032 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2546001035015 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27767771 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56618504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚183614210610539 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102958076 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2555566286336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102958076 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187166856133632 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56592136 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2675303970304 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 $ _timestamp‚1.788498495894656e+09 _step‚181 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41262133 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚187166838130688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚189722404417024 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102984446 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46365942 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46365940 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13494364 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚183613285595975 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102984446 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 trainer/global_step‚2249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚187166838130688 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2556476270080 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56618504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2546909010219 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41262763 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocate3ê‡ d‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218284032 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2546001035015 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27767771 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56618504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚183614210610539 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102958076 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2555566286336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102958076 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187166856133632 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56592136 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2675303970304 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 $ _timestamp‚1.788498495894656e+09 _step‚181 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41262133 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚187166838130688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚189722404417024 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102984446 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46365942 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46365940 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13494364 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚183613285595975 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102984446 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 trainer/global_step‚2249µi<(í2ë- _wandbruntime‚2553  _runtime‚2553óFYìÓ 0loss/FRt2/assignment_loss‚3.2838427156209944 /loss/FRt4/assignment_loss‚4.145860862731934 %loss/total_loss‚46.63318487113572 _runtime‚2553.675557265 2loss/FBt3/assignment_loss‚0.003956822833667199 /loss/FRt3/detection_loss‚1.6066984849480481 0loss/SRqqt4/detection_loss‚2.588481929372339 /loss/FBt1/detection_loss‚0.8152345358859747 trainer/global_step‚2249 2loss/SRqqt2/assignment_loss‚1.9249650189543472 2loss/SRqqt3/assignment_loss‚1.2722239995763327 0loss/FBt2/assignment_loss‚0.2690194533455984 0loss/FBt4/assignment_loss‚0.3801015883433365 /loss/FRt2/detection_loss‚2.3416554920375345 .loss/FBt3/detection_loss‚3.855768750111262 0loss/FRt1/assignment_loss‚2.1509242883095374 1loss/SRqqt4/assignment_loss‚2.411262009511976 0loss/SRqqt1/detection_loss‚2.822008723562414 epoch‚1 $ _timestamp‚1.788498495898479e+09 _step‚182 /loss/FRt1/detection_loss‚1.7631941027122173 1loss/SRqqt3/detection_loss‚1.0873325768237312 0loss/FRt3/assignment_loss‚1.7557004772699796 2loss/SRqqt1/assignment_loss‚1.3606594211675904 1loss/FBt1/assignment_loss‚0.13949536429109646 .loss/FRt4/detection_loss‚3.579282408952713 0loss/SRqqt2/detection_loss‚2.453162733246298 .loss/FBt2/detection_loss‚2.362930187157222 /loss/FBt4/detection_loss‚2.2594229243695736!'÷Ú^íØ .loss/FRt4/detection_loss‚3.579282408952713 0loss/SRqqt2/detection_loss‚2.453162733246298 .loss/FBt2/detection_loss‚2.362930187157222 /loss/FBt4/detection_loss‚2.2594229243695736 0loss/FRt2/assignment_loss‚3.2838427156209944 /loss/FRt4/assignment_loss‚4.145860862731934 %loss/total_loss‚46.63318487113572 _runtime‚2553.675557265 2loss/FBt3/assignment_loss‚0.003956822833667199 /loss/FRt3/detection_loss‚1.6066984849480481 0loss/SRqqt4/detection_loss‚2.588481929372339 /loss/FBt1/detection_loss‚0.8152345358859747 trainer/global_step‚2249 2loss/SRqqt2/assignment_loss‚1.9249650189543472 2loss/SRqqt3/assignment_loss‚1.2722239995763327 0loss/FBt2/assignment_loss‚0.2690194533455984 0loss/FBt4/assignment_loss‚0.3801015883433365 /loss/FRt2/detection_loss‚2.3416554920375345 .loss/FBt3/detection_loss‚3.855768750111262 0loss/FRt1/assignment_loss‚2.1509242883095374 1loss/SRqqt4/assignment_loss‚2.411262009511976 0loss/SRqqt1/detection_loss‚2.822008723562414 epoch‚1 $ _timestamp‚1.788498495898479e+09 _step‚182 /loss/FRt1/detection_loss‚1.7631941027122173 1loss/SRqqt3/detection_loss‚1.0873325768237312 0loss/FRt3/assignment_loss‚1.7557004772699796 2loss/SRqqt1/assignment_loss‚1.3606594211675904 1loss/FBt1/assignment_loss‚0.13949536429109646¶$q“î‚(:ˆ ˤéԤΎm gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚173.059' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.26475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žeN ]ï‚(:Ò ˤéÔù¢Èm) disk./.usagePercent‚83.73823235681208 disk.dm-0.in‚ 65.56640625$ memory_percent‚99.96764659881592 disk.md0p1.out‚ 269.421875 disk.dm-0.out‚29320.86328125 disk.nvme0n1.in‚ 56.55859375 disk.md0.in‚ 1.94140625 disk.sdb2.in‚ 0.26953125 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sdb.out‚ 280.263671875 disk.rbd0.in‚0 disk.rbd0.out‚ 0.57080078125 network.recv‚881545 disk.md0p1.in‚ 1.94140625 disk.sdb.in‚ 0.26953125# disk.nvme0n1.out‚12703.29296875 disk.nvme1n1.in‚ 9.0078125 disk.rbd2.in‚0 disk.sdb2.out‚ 280.263671875 disk.sda.in‚1.671875 network.sent‚ 4.462449e+06 disk.sdb1.out‚0 disk.sda2.in‚1.671875 disk.sda.out‚ 280.263671875 disk.sda1.in‚0& proc.memory.availableMB‚ 5.30078125 disk.sdb1.in‚0" disk.nvme1n1.out‚ 16617.5703125 disk.md0.out‚ 269.421875# proc.memory.rssMB‚ 2992.33203125 cpu‚25.12884805923838 disk.rbd2.out‚ 0.845703125$ disk./.usageGB‚2994.884796142578 disk.sda2.out‚ 280.263671875* proc.memory.percent‚18.263745307922363Cdm“ð‚(:ˆ Ú¤éÔÎÆÎk gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨}aU[ñ‚(:Ð Ú¤éÔÎð„m network.sent‚ 4.570067e+06 disk.nvme1n1.in‚ 9.0078125 disk.sdb2.in‚ 0.26953125 disk.sda.in‚1.671875 disk.sda.out‚ 281.69140625% proc.memory.availableMB‚ 4.9296875 proc.cpu.threads‚17 network.recv‚887631% disk./.usageGB‚2994.8284797668457 disk.md0.in‚ 1.94140625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.out‚ 29513.640625 disk.nvme0n1.in‚ 56.55859375 disk.md0p1.in‚ 1.94140625 disk.sdb.out‚ 281.69140625 disk.sda2.out‚ 281.69140625$ memory_percent‚99.96991157531738) disk./.usagePercent‚83.73665772737637# disk.nvme1n1.out‚16618.11328125 disk.md0.out‚ 270.7890625 disk.rbd0.out‚ 0.57080078125# disk.nvme0n1.out‚12895.52734375 cpu‚25.128020648566512 disk.rbd2.out‚ 0.845703125 disk.dm-0.in‚ 65.56640625 disk.sdb.in‚ 0.26953125# proc.memory.rssMB‚ 2992.33203125 disk.rbd2.in‚0 disk.sdb2.out‚ 281.69140625 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda2.in‚1.671875 disk.sda1.out‚0 disk.md0p1.out‚ 270.7890625* proc.memory.percent‚18.263745307922363Nüó “ò‚(:ˆ é¤éÔ›Õ»k gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚177.411' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.35275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í ²ìMó‚(: é¤éÔ›±£m cpu‚25.127283263112734 disk.nvme1n1.in‚ 9.0078125 disk.sdb2.in‚ 0.26953125$ proc.memory.availableMB‚4.796875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb.out‚ 283.44921875$ memory_percent‚99.97072219848633 network.recv‚891588 disk.sda.out‚ 283.44921875 disk.nvme0n1.in‚ 56.55859375# proc.memory.rssMB‚ 2992.33203125 proc.cpu.threads‚17 disk.md0.in‚ 1.94140625 disk.sdb2.out‚ 283.44921875 disk.sdb1.in‚0 disk.dm-0.in‚ 65.56640625 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sdb.in‚ 0.26953125$ disk./.usageGB‚2995.414260864258 disk.md0p1.out‚272.5# disk.nvme0n1.out‚12969.20703125) disk./.usagePercent‚83.75303641202844" disk.nvme1n1.out‚ 16661.4140625 disk.rbd2.in‚0 disk.rbd2.out‚ 0.845703125 disk.rbd0.out‚ 0.57080078125 disk.md0p1.in‚ 1.94140625* proc.memory.percent‚18.263745307922363 disk.sda2.in‚1.671875 disk.sda2.out‚ 283.44921875 network.sent‚ 4.57535e+06 disk.md0.out‚272.5 disk.sda.in‚1.671875 disk.dm-0.out‚29630.62109375ÈðL2ô- _wandbruntime‚2607  _runtime‚2607”Ñ¥ÏÇõÁ % _timestamp‚1.7884985500572672e+09 _runtime‚2607.834488118 _step‚183 & lr-AdamW/pg1‚0.0010265115344368664 & lr-AdamW/pg2‚0.0010265115344368664 trainer/global_step‚2299Š7ÌöÆ & lr-AdamW/pg1‚0.0010265115344368664 & lr-AdamW/pg2‚0.0010265115344368664 trainer/global_step‚2299 % _timestamp‚1.7884985500572672e+09 _runtime‚2607.834488118 _step‚183·¼ti2÷- _wandbruntime‚2607  _runtime‚2607MXÈêò'øìO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2600761016845 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2611442221568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 105192331 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57838549 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248528896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2610532251136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224588800 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚193777633560576 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚211084966557184 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _step‚184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚191166191339008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28358956 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47353780 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28358954 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193776705587200 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚187540832724817 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚184939163732976 % _timestamp‚1.7884985500583913e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚191166191339008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚213818044477440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚184939146693616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _runtime‚2607.835790084 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2611442221568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚213817795948544 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚193777633560576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚191166173336064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2610532251136 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚191166173336064 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57812207 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105165987 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2601668991841 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57812207 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42141020 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42140397 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13781443 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57838549 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185„›—ÆôùñO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2611442221568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 105192331 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57838549 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248528896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2610532251136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224588800 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚193777633560576 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚211084966557184 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _step‚184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚191166191339008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28358956 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47353780 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28358954 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193776705587200 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚187540832724817 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚184939163732976 % _timestamp‚1.7884985500583913e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚191166191339008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚213818044477440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚184939146693616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _runtime‚2607.835790084 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2611442221568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚213817795948544 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚193777633560576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401Úª+Ü 142784 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚191166173336064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2610532251136 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚191166173336064 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57812207 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105165987 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2601668991841 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57812207 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42141020 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42140397 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13781443 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57838549 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2600761016845 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432¸Ží”2ú- _wandbruntime‚2608  _runtime‚2608§…q·ú&ûôM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚191247810424832 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚193860374119936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105210978 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240126976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚187620042578283 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚191247792421888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2611653711360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42158233 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚213909451501056 trainer/global_step‚2299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28371021 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚191247810424832 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57837038 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚211175216201728 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 # _timestamp‚1.78849855111394e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47373940 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47373942 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47373942 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211175192261632 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚185018164288160 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13787834 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚213909211374080 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57837038 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28371019 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42158855 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2602786265327 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216186880 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47373940 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚187620967592847 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57863406 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2612563695104 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57863406 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2601878290123 _runtime‚2608.891775734 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105237348 _step‚185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2734019112448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513Ç|ÿ&üùM WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105210978 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28371019 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42158855 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2602786265327 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216186880 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47373940 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚187620967592847 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57863406 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2612563695104 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57863406 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2601878290123 _runtime‚2608.891775734 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105237348 _step‚185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2734019112448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚191247810424832 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚193860374119936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105210978 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240126976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚187620042578283 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚191247792421888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2611653711360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42158233 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚213909451501056 trainer/global_step‚2299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28371021 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚191247810424832 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57837038 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚211175216201728 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 # _timestamp‚1.78849855111394e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47373940 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47373942 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47373942 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211175192261632 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚185018164288160 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13787834 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚213909211374080 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57837038 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 BÔ¯¥ŒêO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59083451 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48361780 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚195247127627264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2791708597248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59057109 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28962206 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚195247127627264 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚195247145630208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14072976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2667529646592 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2666619676160 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250626048 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 trainer/global_step‚2349 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚197913747303424 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2666619676160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚197914675276800 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚2665.412056177 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚195247145630208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 # _timestamp‚1.78849860763474e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚218388211002880 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚197914675276800 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚215596275719680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚188890043460176 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48361782 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚191546664692769 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43034545 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _step‚188 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚215596251779584 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚188890026420816 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356µË_õ'ïO O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59083451 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48361780 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚195247127627264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2791708597248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59057109 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28962206 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚195247127627264 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚195247145630208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14072976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2667529646592 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2666619676160 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250626048 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 trainer/global_step‚2349 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚197913747303424 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2666619676160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚197914675276800 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚2665.412056177 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚195247145630208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 # _timestamp‚1.78849860763474e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚218388211002880 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚197914675276800 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚215596275719680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚188890043460176 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48361782 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚191546664692769 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43034545 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _step‚188 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚215596251779584 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚188890026420816 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784¼ j½Ú2Ž- _wandbruntime‚2666  _runtime‚2666–:¼aü&öM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2793102365696 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2792875693056 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚191627724575155 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14078811 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚612 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚195328764716032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2668651120128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226672640 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2657755545231 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _step‚189 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2667741136384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28974271 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59081940 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚218479353177088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43052468 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚195328764716032 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43053082 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14078199 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚197996487849472 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚218479603789824 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107463880 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48381942 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2668651120128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107463880 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107490250 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107490250 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250612736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚215686477484032 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚195328746713088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59108308 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48381940 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14078811 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚612 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚195328764716032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2668651120128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226672640 XFDeviceStatsM0égÑ’onitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2657755545231 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _step‚189 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2667741136384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28974271 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59081940 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚218479353177088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43052468 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚195328764716032 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43053082 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14078199 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚197996487849472 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚218479603789824 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107463880 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48381942 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2668651120128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107463880 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107490250 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107490250 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250612736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚215686477484032 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚195328746713088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59108308 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48381940 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚192840906148016 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2713423502057 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚199328099921408 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2723617071616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚202051716993024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218297344 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚195554346689433 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚220107537001984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚192840923187376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚202050789019648 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚199328099921408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚220107560942080 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _runtime‚ 2717.21365058 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49369782 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43930538 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚192 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚199328081918464 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚195553421675077 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2712515527061 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2722707101184 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚192840906148016 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2713423502057 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚199328099921408 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2723617071616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚202051716993024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218297344 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚195554346689433 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚220107537001984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚192840923187376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚202050789019648 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚199328099921408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚220107560942080 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _runtime‚ 2717.21365058 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocÛ9PТated‚49369782 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43930538 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚192 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚199328081918464 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚195553421675077 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2712515527061 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2722707101184 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49389940 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109743152 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2724738545152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚192919923742560 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14370323 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚202134457552384 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43947842 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2851720760320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚195633556542899 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14370965 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 $ _timestamp‚1.788498660402661e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2723828561408 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2714540775543 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60353210 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚202134457552384 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49389942 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109716782 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚192919940781920 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60326842 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚220197762706432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29577519 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚202134457552384 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43947842 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2851720760320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚195633556542899 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14370965 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 $ _timestamp‚1.788498660402661e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2723828561408 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2714540775543 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60353210 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚202134457552384 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49389942 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109716782 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚192919940781920 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60326842 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚220197762706432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29577519 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49389940 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109743152 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2724738545152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚192919923742560 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14370323 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚199571031015363 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13205766144 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 111722333 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚224578935522304 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚44 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 2897144174080 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 2768944711168 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚61211189 % _timestamp‚1.7884987413671508e+09 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 701041152 _runtime‚ 2799.14509058 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 701041152 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13684416513 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1185 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20711 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32247 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚220 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 172599808 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 2768261672960 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚199571729122163 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1187728385 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1185 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚218 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚227476276236288 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚196813136637728 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚196813119598368 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 681067440 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 _step‚195ÌOÑì]*´×T QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚61211189 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚206306591706624 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1185 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20711 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 14135737344 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 698106800 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚570 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚14739129 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 2897316773888 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚206305890665472 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 111743044 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1231 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚615 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 14135737344 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚203537646995456 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚203537628992512 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 2757911416995 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14401142784 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚227476079696384 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14401142784 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚45002138 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189767168 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 111722333 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚570 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚570 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚199571031015363 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13205766144 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 111722333 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚224578935522304 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚44 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 2897144174080 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 2768944711168 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚61211189 % _timestamp‚1.7884987413671508e+09 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 701041152 _runtime‚ 2799.14509058 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 701041152 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13684416513 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1185 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20711 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32247 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚220 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 172599808 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 2768261672960 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚199571729122163 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1187728385 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1185 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚218 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚227476276236288 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚196813136637728 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚196813119598368 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 681067440 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 _step‚195 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20709 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚50511144 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20709ÊÙyO2µ- _wandbruntime‚2800  _runtime‚2800ÛZÖHÌ"¶ÜR ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1185 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32004 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 2769434562048 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚61240712 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14401142784 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14401142784 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚206361342464512 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚61220003 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698106800 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1268 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 14135737344 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189767168 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚206360641423360 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 2768751523840 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 409073152 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 2759080695319 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 172599808 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 701041152 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 2768751523840 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 111767334 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3725 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚50526622 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚61240712 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 2769434562048 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 683038208 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 2897806624768 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚196863562742912 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788498742457032e+09 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚45017080 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚206361342464512 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚14744291 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13205766144 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135737344 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1185 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 111746623 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 111746623 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚14744510 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚224635203076096 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚615 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚203591907902464 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚219 _runtime‚2800.235001094 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20709 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13684416513 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 683038208 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 701041152 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚206360641423360 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32247 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20709 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20711 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚570 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚30272789 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32247 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 196539904 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚221 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚50526620 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚45017301 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚615 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚203591907902464 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚227532837101056 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1187728385 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚196863545703552 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 2758399627879 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚44 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1268 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 14135737344 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189767168 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚206360641423360 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 2768751523840 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 409073152 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 2759080695319 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 172599808 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 701041152 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 2768751523840 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 111767334 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3725 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚50526622 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚61240712 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 2769434562048 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 683038208 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 2897806624768 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚196863562742912 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788498742457032e+09 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚45017080 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚206361342464512 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚14744291 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13205766144 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135737344 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1185 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 111746623 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 111746623 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚14744510 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚224635203076096 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚615 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚203591907902464 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚219 _runtime‚2800.235001094 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20709 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13684416513 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 683038208 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 701041152 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚206360641423360 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32247 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20709 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20711 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚570 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚30272789 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32247 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 196539904 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚221 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚50526620 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚45017301 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚615 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚203591907902464 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚227532837101056 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1187728385 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚196863545703552 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 2758399627879 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚44 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 B&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9872086048126221 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9864341616630554 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.23514266312122345 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2253306806087494 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 trainer/global_step‚2417 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_4‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.010728923603892326 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18341979384422302 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.3414141535758972 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40035292506217957 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.35513392090797424 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.990949809551239 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 particle/accuracy_4_of_7‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.33343908190727234 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.09416338056325912 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8859649300575256 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6896177530288696 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.47143715620040894 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3415816128253937 *jet/accuracy_2_of_6‚0.4166666865348816 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666269302368 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.3684210479259491 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 _runtime‚2815.347932077 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26675382256507874 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.40410372614860535 particle/accuracy_1_of_2‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1347254514694214 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4667789936065674 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08676642179489136 *jet/accuracy_2_of_3‚0.2111111879348755 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9137115478515625 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.07895144820213318 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9884469509124756 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14591693878173828 particle/accuracy_1_of_3‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09751541912555695 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23601758480072021 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9993650913238525 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.025383083149790764 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.007709750905632973 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 particle/accuracy_2_of_5‚1 particle/accuracy_4_of_6‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_7‚0 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.375396728515625 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2224845141172409 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2083333283662796 6 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.183737650513649 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.2977272570133209 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9990973472595215 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.2382424771785736 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8163588047027588 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18682751059532166 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.00515872985124588 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3460237383842468 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.34492117166519165 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9890351295471191 +jet/accuracy_2_of_4‚0.32947081327438354 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.11363637447357178 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.3749999701976776 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.44366422295570374 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.35031580924987793 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.1666666716337204 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1805555671453476 +jet/accuracy_3_of_6‚0.07692307978868484 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_4‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9993650913238525 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.30729997158050537 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2742410898208618 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.4863094985485077 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3953619599342346 +jet/accuracy_3_of_5‚0.06352406740188599 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.33495789766311646 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03488372266292572 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9062500596046448 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01024913601577282 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3967784643173218 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3811032474040985 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.20150762796401978 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4503476321697235 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3544677495956421 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.31480464339256287 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983749389648438 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.021551724523305893 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.3902294635772705 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 % _timestamp‚1.7884987575681055e+09 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.43282702565193176 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.22602339088916779 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.006784434430301189 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.21659521758556366 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8751716017723083 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.125 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.29410043358802795 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9892389178276062 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 6 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.990007758140564 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49000000953674316 jet/accuracy_6_of_7‚0 _step‚197 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1422642171382904 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18827810883522034 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.015086236409842968 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.02140614576637745 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_6‚0 particle/accuracy_6_of_6‚1 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.22736626863479614 jet/accuracy_5_of_5‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.16230271756649017 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03689800947904587 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9984127283096313 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.25540250539779663 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 jet/accuracy_3_of_7‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.383213073015213 *loss/val_total_loss‚45.463406166242805 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5438596606254578 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9915786385536194 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.624440610408783 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.27504804730415344 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9861111640930176 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.2553376853466034 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9938936829566956 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.25958606600761414 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 jet/accuracy_5_of_7‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1871185153722763 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.38887229561805725 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 +jet/accuracy_4_of_6‚0.01923076994717121 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.22499088943004608 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.24411919713020325 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37428542971611023 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.2845690846443176 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 *jet/accuracy_2_of_5‚0.3974078297615051 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.36994048953056335 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1666666716337204 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9863860607147217 @&Purity/0FRt4LAóvHSRqqt0FBt/event_proportion‚0.0004340277810115367 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3600088655948639 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.01021033339202404 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21453909575939178 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 )jet/accuracy_1_of_3‚0.812850296497345 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.140625 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9897581934928894 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9943121671676636 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3304356336593628 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03258928656578064 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_4‚0.05799978971481323 particle/accuracy_3_of_3‚1 particle/accuracy_2_of_7‚1 particle/accuracy_3_of_7‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9980646371841431 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.38271602988243103 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 particle/accuracy_1_of_7‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2047479748725891 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1240251287817955 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.48841118812561035 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.006453784182667732 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 particle/accuracy_6_of_7‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01601114124059677 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 particle/accuracy_2_of_6‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22824349999427795 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.33040541410446167 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9944241046905518 particle/accuracy_5_of_5‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 particle/accuracy_1_of_6‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7575758099555969 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 jet/accuracy_1_of_7‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11974203586578369 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.999630868434906 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5167015790939331 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.32648804783821106 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24034203588962555 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4687499701976776 particle/accuracy_7_of_7‚1 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.33373603224754333 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.2261369377374649 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.22642643749713898 ,jet/accuracy_4_of_4‚0.004317838232964277 *jet/accuracy_1_of_5‚0.9126471281051636 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3740741014480591 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2735448479652405 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2916666865348816 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.2525072395801544 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1507936418056488 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5579860806465149 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1199050024151802 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9947635531425476 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 & Purity/1FRt1SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.33306530117988586 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.45476657152175903 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1666666865348816 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.40625 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2420259416103363 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 jet/accuracy_1_of_6‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4778963029384613 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4222085177898407 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.40019381046295166 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9942857027053833 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.34899404644966125 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.49558231234550476 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.25488507747650146 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.024376271292567253 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.42298853397369385 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2629310190677643 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3598484992980957 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.1666666716337204 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.33868011832237244 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.017153393477201462 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05263157933950424 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.41813844442367554 particle/accuracy_3_of_5‚1 (particle/accuracy‚0.8462663292884827 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12411347031593323 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5383880138397217 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.4117647111415863 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961150288581848 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.19339081645011902 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 (particle/f_score‚0.37347444891929626 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4748347997665405 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3480808138847351 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9126984477043152 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5312728881835938 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.2614942789077759 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.40086206793785095 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4305556118488312 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7916666865348816 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.2708333432674408 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14276696741580963 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.35729166865348816 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *particle/specificity‚0.948553204536438 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3575213849544525 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23400317132472992 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2280471920967102 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 particle/accuracy_5_of_7‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9444444179534912 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9870763421058655 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06418003141880035 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5272465348243713 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1782493144273758 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.21370455622673035 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 +particle/sensitivity‚0.2936171591281891 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9926760792732239 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 jet/accuracy_2_of_7‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1521868258714676 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_4‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 epoch‚1 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3611111044883728 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23769062757492065 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.42105263471603394 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.550595223903656 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5023148059844971 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1666666716337204 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4132128357887268 7"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.45333331823349 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.13815215229988098 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.33783069252967834 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.3410494029521942 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9629629850387573 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.996597170829773 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.46400219202041626 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9876013398170471 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3812049329280853 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.33611631393432617 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 6validation_average_jet_accuracy‚0.3768734037876129 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23947803676128387 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.31707754731178284 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.478123277425766 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.012698412872850895 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9923065304756165 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.23333333432674408 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.3333333134651184 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3333333134651184 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3683956563472748 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2260943055152893 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.3749999701976776.=îÔ|7¼Ť 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4748347997665405 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3480808138847351 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9126984477043152 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5312728881835938 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.2614942789077759 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.40086206793785095 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4305556118488312 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7916666865348816 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.2708333432674408 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14276696741580963 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.35729166865348816 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *particle/specificity‚0.948553204536438 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3575213849544525 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23400317132472992 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2280471920967102 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 particle/accuracy_5_of_7‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9444444179534912 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9870763421058655 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06418003141880035 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5272465348243713 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1782493144273758 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.21370455622673035 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 +particle/sensitivity‚0.2936171591281891 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9926760792732239 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 jet/accuracy_2_of_7‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1521868258714676 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_4‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 epoch‚1 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3611111044883728 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23769062757492065 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.42105263471603394 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.550595223903656 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5023148059844971 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1666666716337204 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4132128357887268 7"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.45333331823349 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.13815215229988098 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.33783069252967834 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.3410494029521942 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9629629850387573 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.996597170829773 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.46400219202041626 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9876013398170471 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3812049329280853 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.33611631393432617 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 6validation_average_jet_accuracy‚0.3768734037876129 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23947803676128387 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.31707754731178284 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.478123277425766 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.012698412872850895 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9923065304756165 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.23333333432674408 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.3333333134651184 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3333333134651184 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3683956563472748 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2260943055152893 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.3749999701976776 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.020440252497792244 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1805555671453476 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.24228395521640778 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.30729997158050537 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21162360906600952 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.2646198570728302 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.001717256149277091 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1071428582072258 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.2708333432674408 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.6776877045631409 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19752970337867737 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9879858493804932 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3118685781955719 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3454959988594055 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5280464887619019 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.24401655793190002 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14276696741580963 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9872086048126221 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9864341616630554 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.23514266312122345 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2253306806087494 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 trainer/global_step‚2417 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_4‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.010728923603892326 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18341979384422302 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.3414141535758972 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40035292506217957 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.35513392090797424 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.990949809551239 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 particle/accuracy_4_of_7‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.33343908190727234 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.09416338056325912 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8859649300575256 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6896177530288696 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.47143715620040894 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3415816128253937 *jet/accuracy_2_of_6‚0.4166666865348816 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666269302368 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.3684210479259491 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 _runtime‚2815.347932077 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26675382256507874 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.40410372614860535 particle/accuracy_1_of_2‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1347254514694214 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4667789936065674 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08676642179489136 *jet/accuracy_2_of_3‚0.2111111879348755 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 7 Purity/*FRg¬ivÐZt*SRqqt3FBt/FBt_purity‚0.9137115478515625 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.07895144820213318 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9884469509124756 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14591693878173828 particle/accuracy_1_of_3‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09751541912555695 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23601758480072021 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9993650913238525 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.025383083149790764 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.007709750905632973 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 particle/accuracy_2_of_5‚1 particle/accuracy_4_of_6‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_7‚0 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.375396728515625 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2224845141172409 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2083333283662796 6 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.183737650513649 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.2977272570133209 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9990973472595215 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.2382424771785736 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8163588047027588 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18682751059532166 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.00515872985124588 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3460237383842468 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.34492117166519165 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9890351295471191 +jet/accuracy_2_of_4‚0.32947081327438354 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.11363637447357178 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.3749999701976776 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.44366422295570374 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.35031580924987793 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.1666666716337204 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1805555671453476 +jet/accuracy_3_of_6‚0.07692307978868484 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_4‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9993650913238525 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.30729997158050537 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2742410898208618 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.4863094985485077 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3953619599342346 +jet/accuracy_3_of_5‚0.06352406740188599 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.33495789766311646 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03488372266292572 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9062500596046448 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01024913601577282 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3967784643173218 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3811032474040985 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.20150762796401978 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4503476321697235 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3544677495956421 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.31480464339256287 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983749389648438 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.021551724523305893 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.3902294635772705 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 % _timestamp‚1.7884987575681055e+09 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.43282702565193176 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.22602339088916779 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.006784434430301189 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.21659521758556366 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8751716017723083 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.125 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.29410043358802795 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9892389178276062 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 6 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.990007758140564 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49000000953674316 jet/accuracy_6_of_7‚0 _step‚197 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1422642171382904 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18827810883522034 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.015086236409842968 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.02140614576637745 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_6‚0 particle/accuracy_6_of_6‚1 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.22736626863479614 jet/accuracy_5_of_5‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.16230271756649017 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03689800947904587 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9984127283096313 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.25540250539779663 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 jet/accuracy_3_of_7‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.383213073015213 *loss/val_total_loss‚45.463406166242805 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5438596606254578 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9915786385536194 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.624440610408783 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.27504804730415344 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9861111640930176 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.2553376853466034 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9938936829566956 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.25958606600761414 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 jet/accuracy_5_of_7‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1871185153722763 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.38887229561805725 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 +jet/accuracy_4_of_6‚0.01923076994717121 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.22499088943004608 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.24411919713020325 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37428542971611023 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.2845690846443176 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 *jet/accuracy_2_of_5‚0.3974078297615051 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.36994048953056335 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1666666716337204 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9863860607147217 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3600088655948639 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.01021033339202404 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21453909575939178 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 )jet/accuracy_1_of_3‚0.812850296497345 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.140625 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9897581934928894 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9943121671676636 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3304356336593628 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03258928656578064 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_4‚0.05799978971481323 particle/accuracy_3_of_3‚1 particle/accuracy_2_of_7‚1 particle/accuracy_3_of_7‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9980646371841431 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.38271602988243103 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 particle/accuracy_1_of_7‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2047479748725891 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1240251287817955 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.48841118812561035 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.006453784182667732 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 particle/accuracy_6_of_7‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01601114124059677 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 particle/accuracy_2_of_6‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22824349999427795 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.33040541410446167 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9944241046905518 particle/accuracy_5_of_5‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 particle/accuracy_1_of_6‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7575758099555969 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 jet/accuracy_1_of_7‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11974203586578369 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.999630868434906 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5167015790939331 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.32648804783821106 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24034203588962555 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4687499701976776 particle/accuracy_7_of_7‚1 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.33373603224754333 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.2261369377374649 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.22642643749713898 ,jet/accuracy_4_of_4‚0.004317838232964277 *jet/accuracy_1_of_5‚0.9126471281051636 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3740741014480591 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2735448479652405 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2916666865348816 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.2525072395801544 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1507936418056488 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5579860806465149 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1199050024151802 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9947635531425476 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 & Purity/1FRt1SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.33306530117988586 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.45476657152175903 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1666666865348816 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.40625 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2420259416103363 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 jet/accuracy_1_of_6‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4778963029384613 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4222085177898407 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.40019381046295166 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9942857027053833 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.34899404644966125 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.49558231234550476 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.25488507747650146 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.024376271292567253 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.42298853397369385 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2629310190677643 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3598484992980957 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.1666666716337204 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.33868011832237244 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.017153393477201462 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05263157933950424 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.41813844442367554 particle/accuracy_3_of_5‚1 (particle/accuracy‚0.8462663292884827 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12411347031593323 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5383880138397217 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.4117647111415863 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961150288581848 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.19339081645011902 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 (particle/f_score‚0.37347444891929626Åo“h½‚(:„ ʦéÔˆŠ¶m gpu.0.gpu‚6.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚32.0 gpu.0.powerWatts‚86.17' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚21.5425 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãp%Ü^¾‚(:Ó ʦéÔÞ«Éo disk.sda2.in‚1.671875 disk.nvme0n1.in‚ 56.56640625 disk.nvme1n1.in‚ 9.0078125 disk.rbd0.in‚0' proc.memory.availableMB‚ 912.0546875 disk.md0.in‚ 1.94140625! disk.sdb2.out‚308.24560546875* proc.memory.percent‚20.862197875976562 proc.cpu.threads‚17" disk./.usageGB‚2996.0283203125 disk.sdb.out‚308.24560546875 disk.rbd2.out‚ 0.845703125 disk.sdb2.in‚ 0.26953125 disk.sda.in‚1.671875 disk.sdb1.in‚0 disk.sdb.in‚ 0.26953125# disk.nvme0n1.out‚15062.73046875 disk.md0.out‚ 296.3046875 network.sent‚ 5.08175e+06) disk./.usagePercent‚83.77020577120514 disk.sdb1.out‚0 disk.dm-0.in‚ 65.57421875! disk.sda2.out‚308.24560546875# disk.nvme1n1.out‚17434.78515625 disk.sda.out‚308.24560546875 disk.sda1.in‚0 disk.md0p1.in‚ 1.94140625 disk.rbd2.in‚0 disk.rbd0.out‚ 0.783203125$ memory_percent‚94.43325996398926 cpu‚25.21347561151916 network.recv‚961322 disk.dm-0.out‚ 32497.515625 disk.sda1.out‚0 proc.memory.rssMB‚ 3418.0625 disk.md0p1.out‚ 296.3046875×.Q/`¿‚(:Õ Ù¦éÔÄõ“m cpu‚25.175252727264265) disk./.usagePercent‚83.75166678413547# disk.nvme1n1.out‚17435.05078125 disk.md0.in‚ 1.94140625) proc.memory.availableMB‚ 1317.15234375 disk.sda1.out‚0 disk.dm-0.out‚34024.73828125 disk.sdb.in‚ 0.26953125 disk.md0.out‚ 297.87109375 disk.rbd2.in‚0 network.sent‚ 5.190908e+06 disk.sdb2.in‚ 0.26953125 disk.sda.in‚1.671875 disk.sdb1.out‚0 disk.md0p1.in‚ 1.94140625 disk.rbd2.out‚ 0.845703125* proc.memory.percent‚18.407344818115234 disk.sdb2.out‚309.8720703125 disk.sda.out‚309.8720703125 disk.sda1.in‚0 disk.nvme0n1.out‚ 16589.6875# memory_percent‚91.9607400894165 network.recv‚966452# disk./.usageGB‚2995.36527633667 disk.rbd0.in‚0 disk.md0p1.out‚ 297.87109375 disk.dm-0.in‚ 68.61328125! proc.memory.rssMB‚ 3015.859375 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.sdb.out‚309.8720703125 disk.sda2.out‚309.8720703125 disk.rbd0.out‚ 0.783203125 disk.sda2.in‚1.671875 disk.nvme0n1.in‚ 58.9921875 disk.nvme1n1.in‚ 9.62109375?ëU‘À‚(:† Ù¦éÔÁÑêm gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚179.856' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.'°ã’Á‚(:‡ è¦éÔû•Øm gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²§U\‚(:Ñ è¦éÔ«œ›n disk.nvme1n1.in‚ 9.62109375! disk.nvme1n1.out‚ 17435.546875 disk.md0.in‚ 1.94140625 disk.rbd2.in‚0 disk.sda2.out‚311.8662109375 proc.cpu.threads‚17 disk.sdb2.out‚311.8662109375 disk.sda1.in‚0* proc.memory.percent‚18.395352363586426 network.sent‚ 5.197035e+06 disk.md0.out‚ 299.8046875 disk.sda.out‚311.8662109375 disk.sda1.out‚0 disk.sdb.out‚311.8662109375' proc.memory.availableMB‚ 1318.515625# proc.memory.rssMB‚ 3013.89453125 disk.nvme0n1.out‚ 16875.6875 disk.sdb1.in‚0 disk.md0p1.in‚ 1.94140625 disk.sda2.in‚1.671875 disk.sdb2.in‚ 0.26953125 disk.md0p1.out‚ 299.8046875 disk.sdb.in‚ 0.26953125 disk.sda.in‚1.671875 disk.rbd0.in‚0 disk.rbd0.out‚ 0.783203125 disk.sdb1.out‚0 disk.dm-0.in‚ 68.61328125 disk.dm-0.out‚ 34311.234375$ memory_percent‚91.95241928100586 network.recv‚970987% disk./.usageGB‚2995.4375343322754 disk.nvme0n1.in‚ 58.9921875 disk.rbd2.out‚ 0.845703125 cpu‚25.17458001324568) disk./.usagePercent‚83.75368714793493‡‘2Ã- _wandbruntime‚2861  _runtime‚2861‹€¼ÜÇÄÁ trainer/global_step‚2449 % _timestamp‚1.7884988040175672e+09 _runtime‚2861.795093657 _step‚198 & lr-AdamW/pg1‚0.0010189908614267375 & lr-AdamW/pg2‚0.00101899086142673755PÅÌÅÆ & lr-AdamW/pg1‚0.0010189908614267375 & lr-AdamW/pg2‚0.0010189908614267375 trainer/global_step‚2449 % _timestamp‚1.7884988040175672e+09 _runtime‚2861.795093657 _step‚198ÆŸ:Ê2Æ- _wandbruntime‚2861  _runtime‚2861ÃèrEÇìO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚209637145470976 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚206827491354112 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 E;DeviceStatsMonitor.on_train_b¨//Ùatch_start/allocation.all.peak‚32248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14990987 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚2861.796858056 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2809672119808 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14990354 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 243462656 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2809672119808 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570T¢¾»÷'ÈñO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2799167175231 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30771110 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62130470 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚209638073444864 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51352768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30771112 _step‚199 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62130470 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14990987 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚2861.796858056 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2809672119808 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14990354 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 243462656 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2809672119808 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚209637145470976 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚206827491354112 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2800284448509 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113528257 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚228253903817728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51372928 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚231195668557312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚200047665135712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2801192423717 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 113501886 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62128958 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45779123 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2811703564288 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚206909092436992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚2862.970676787 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2810793580032 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51372930 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113528257 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚202848857559429 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51372930 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30783175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2811703564288 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62155327 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 265292288 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚200047648096352 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14995948 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚2449 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚200047665135712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2801192423717 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 113501886 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62128958 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45779123 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2811703564288 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚206909092436992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚2862.970676787 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2810793580032 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51372930 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113528257 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚202848857559429 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51372930 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30783175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2811703564288 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62155327 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 265292288 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚200047648096352 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14995948 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚2449 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2800284448509 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113528257 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚228253903817728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51372928 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚231195668557312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63349029 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚206774554659347 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232674987275776 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚213775115161088 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚210908427642368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15276754 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2865759544832 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46651114 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31374360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚213774187187200 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213774187187200 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2866669515776 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚203919510229008 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52360768U`.Å%ÜñO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232674987275776 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚213775115161088 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚210908427642368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15276754 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2865759544832 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46651114 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31374360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚213774187187200 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213774187187200 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2866669515776 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚203919510229008 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52360768 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚213775115161088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 115736142 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52360768 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63375372 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚210908445645312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15277367 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚206775479673707 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63349029 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚210908427642368 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976896 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52360770 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63375372 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916992 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63349029 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚206774554659347 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000Ë é7À2Ý- _wandbruntime‚2915  _runtime‚2915†ªÏü&ÞöM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3000074246144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63400229 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2866881005056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52380930 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚210990046728192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚203998527823552 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚206855614541737 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚213856927733248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63373860 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46669478 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚210990064731136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2999849671168 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115754788 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52380930 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚232765212980224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46668867 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115781159 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚210990046728192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2867790989312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15282442 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚203998544862912 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574976 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚611 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚232765189040128 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2857069678825 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63400229 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚210990064731136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _runtime‚2915.863912227 _step‚204 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚206854689527169 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31386425á§)˜'ßûM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2867790989312 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚235765038711296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115781159 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚210990046728192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2867790989312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15282442 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚203998544862912 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574976 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚611 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚232765189040128 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2857069678825 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63400229 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚210990064731136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _runtime‚2915.863912227 _step‚204 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚206854689527169 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31386425 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3000074246144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63400229 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2866881005056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52380930 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚210990046728192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚203998527823552 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚206855614541737 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚213856927733248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63373860 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46669478 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚210990064731136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2999849671168 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115754788 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52380930 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚232765212980224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46668867 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2922756940800 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚214989381933568 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚217912156877312 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚217912156877312 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53368768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2911829660447 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚614 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15564155 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚217911228903424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚214989381933568 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚237186272498176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚210781311641655 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚207870389956208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2549 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2921846969856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 % _timestamp‚1.7884989098896756e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64593931 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚210782236656015 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2922756940800 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47541765 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53368768 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚217911228903424 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚240243527157760 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53368770 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚2967.667097002 _step‚207 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2921846969856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3057254659584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚214989399936512 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64593931 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53368770 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2910921685447 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117989044 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117962699 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚237186272498176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚210781311641655 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚207870389956208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2549 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2921846969856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 % _timestamp‚1.7884989098896756e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64593931 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚210782236656015 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2922756940800 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47541765 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53368768 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚217911228903424 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚240243527157760 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53368770 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚2967.667097002 _step‚207 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2921846969856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3057254659584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚214989399936512 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64593931 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53368770 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2910921685447 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117989044 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117962699 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2922756940800 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2ÏŽ¹™2ï- _wandbruntime‚2968  _runtime‚2968)‚ùü&ðöM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2923878414336 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53388930 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15569796 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64618762 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215071001019392 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64645131 _runtime‚2968.806640624 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097984 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 118034061 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64618762 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31989675 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3058424040960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _step‚208 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53388930 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚215071019022336 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47559471 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚237276474262528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53388928 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚215071001019392 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 118007690 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚215071019022336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118034061 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47560099 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚210862371524045 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚210861446509477 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚217994897436672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2912946933933 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 118007690 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚237276498202624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚2549 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚217994897436672 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚217993969449472 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3058186883072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚240334922243584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚207949424590112 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31989677 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚207949407550752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2922968430080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2912038958725 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 % _timestamp‚1.7884989110284247e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570k^4®(ñûM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚215071019022336 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47559471 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚237276474262528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53388928 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚215071001019392 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 118007690 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚215071019022336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118034061 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47560099 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚210862371524045 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚210861446509477 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚217994897436672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2912946933933 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 118007690 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚237276498202624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚2549 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚217994897436672 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚217993969449472 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3058186883072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚240334922243584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚207949424590112 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31989677 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚207949407550752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2922968430080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2912038958725 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 % _timestamp‚1.7884989110284247e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ;5DeviceStatsMonitor.on_train_batch_en}aÙ d/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2923878414336 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53388930 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15569796 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64618762 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215071001019392 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64645131 _runtime‚2968.806640624 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097984 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 118034061 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64618762 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31989675 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3058424040960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _step‚208 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53388930 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚222048270619648 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54376770 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48430440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48431070 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32580862 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚630 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3115556843520 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚214788068623963 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54376768 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _runtime‚3021.309357503 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚222049198593536 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2978844365824 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65838833 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2977934394880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 % _timestamp‚1.7884989635319893e+09 _step‚211 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚222049198593536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120241946 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244813114564096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2966798940555 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚241697533780480 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚211821286722768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚219070336224768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2978844365824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15850208 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2977934394880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32580860 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚241697557720576 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54376768 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782592 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54376770 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚214788993638323 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚219070354227712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570vH,ý$‚ñO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48431070 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32580862 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚630 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3115556843520 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚214788068623963 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54376768 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _runtime‚3021.309357503 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚222049198593536 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2978844365824 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65838833 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2977934394880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 % _timestamp‚1.7884989635319893e+09 _step‚211 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚222049198593536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120241946 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244813114564096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2966798940555 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚241697533780480 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚211821286722768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚219070336224768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2978844365824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15850208 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2977934394880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32580860 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚241697557720576 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54376768 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782592 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54376770 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚214788993638323 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚219070354227712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65865176 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.smallûªÜáú_pool.current‚ 907975000 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚222048270619648 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54376770 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48430440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0Ó–î…ˆ2ƒ- _wandbruntime‚3022  _runtime‚3022ê%ü&„öM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 % _timestamp‚1.7884989645853472e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65890033 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚222131939152896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 120286963 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65890033 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3116485734912 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚219151955310592 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚222131011165696 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚211900287277952 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48448190 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚241787759484928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚219151973313536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚214869128506353 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120260592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15855265 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚244904245219840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3116718698496 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241787783425024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54396930 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚214868203491785 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120260592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2979055855104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15855897 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2968824189041 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32592927 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚3022.363781642 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54396928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _step‚212 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2979965839360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32592925 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚219151955310592 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54396928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚211900304317312 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚222131939152896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2979055855104 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54396930 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65863664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65863664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2979965839360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120286963 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚244904502123520 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2967916213833 trainer/global_step‚2599 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48448190 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚241787759484928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚219151973313536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚214869128506353 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120260592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15855265 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚244904245219840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3116718698496 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241787783425024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54396930 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚214868203491785 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120260592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2979055855104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15855897 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2968824189041 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32592927 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚3022.363781642 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54396928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _step‚212 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2979965839360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32592925 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚219151955310592 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54396928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚211900304317312 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚222131939152896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2979055855104 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54396930 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65863664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65863664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2979965839360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120286963 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚244904502123520 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2967916213833 trainer/global_step‚2599 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚222131939152896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 120286963 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65890033 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3116485734912 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185Ô w.W2†- _wandbruntime‚3022  _runtime‚3022af'G[‡Õ 1loss/FBt1/assignment_loss‚0.09912469004059528 1loss/SRqqt3/detection_loss‚0.8984299458887266 /loss/FBt1/detection_loss‚0.8353031032495726 /loss/FBt2/detection_loss‚2.3127714185153736 epoch‚2 _runtime‚3022.364667437 2loss/SRqqt3/assignment_loss‚1.2770131278297177 /loss/FRt3/detection_loss‚1.8045224193213643 /loss/FRt1/assignment_loss‚3.014908737914507 /loss/FRt4/assignment_loss‚3.195197860399882 2loss/SRqqt1/assignment_loss‚1.7322848679290876 2loss/SRqqt2/assignment_loss‚1.4109967917736088 1loss/SRqqt4/detection_loss‚2.7627303515161787 .loss/FBt4/detection_loss‚2.738943758110205 .loss/FRt2/detection_loss‚2.487842042557895 trainer/global_step‚2599 0loss/FRt2/assignment_loss‚3.7281323701143263 0loss/FRt3/assignment_loss‚1.6972130943035735 2loss/SRqqt4/assignment_loss‚1.7707204825821377 /loss/FRt4/detection_loss‚3.0027401546637216 /loss/FBt3/detection_loss‚1.9203191369771957 $ _timestamp‚1.788498964587648e+09 1loss/FBt2/assignment_loss‚0.33257323604790573 1loss/FBt3/assignment_loss‚0.06724217626760946 0loss/FBt4/assignment_loss‚0.1857157321435731 .loss/FRt1/detection_loss‚2.047324258162705 1loss/SRqqt1/detection_loss‚1.8279504055778186 %loss/total_loss‚44.00421829787429 0loss/SRqqt2/detection_loss‚2.854218135987009 _step‚213`ŽæP`ˆÚ $ _timestamp‚1.788498964587648e+09 1loss/FBt2/assignment_loss‚0.33257323604790573 1loss/FBt3/assignment_loss‚0.06724217626760946 0loss/FBt4/assignment_loss‚0.1857157321435731 .loss/FRt1/detection_loss‚2.047324258162705 1loss/SRqqt1/detection_loss‚1.8279504055778186 %loss/total_loss‚44.00421829787429 0loss/SRqqt2/detection_loss‚2.854218135987009 _step‚213 1loss/FBt1/assignment_loss‚0.09912469004059528 1loss/SRqqt3/detection_loss‚0.8984299458887266 /loss/FBt1/detection_loss‚0.8353031032495726 /loss/FBt2/detection_loss‚2.3127714185153736 epoch‚2 _runtime‚3022.364667437 2loss/SRqqt3/assignment_loss‚1.2770131278297177 /loss/FRt3/detection_loss‚1.8045224193213643 /loss/FRt1/assignment_loss‚3.014908737914507 /loss/FRt4/assignment_loss‚3.195197860399882 2loss/SRqqt1/assignment_loss‚1.7322848679290876 2loss/SRqqt2/assignment_loss‚1.4109967917736088 1loss/SRqqt4/detection_loss‚2.7627303515161787 .loss/FBt4/detection_loss‚2.738943758110205 .loss/FRt2/detection_loss‚2.487842042557895 trainer/global_step‚2599 0loss/FRt2/assignment_loss‚3.7281323701143263 0loss/FRt3/assignment_loss‚1.6972130943035735 2loss/SRqqt4/assignment_loss‚1.7707204825821377 /loss/FRt4/detection_loss‚3.0027401546637216 /loss/FBt3/detection_loss‚1.9203191369771957ÕÇLÛÇ›‰‚(: œ¨éԴܨm gpu.0.gpu‚8.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚117.81' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.452499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê 6<]Š‚(:Ò œ¨éÔô­ãm disk.sdb.out‚332.2158203125 disk.sda2.in‚1.671875 cpu‚25.164629580010676 disk.sda1.in‚0 disk.md0p1.out‚ 319.34375 proc.cpu.threads‚17# disk.nvme1n1.out‚17565.71484375 disk.rbd2.in‚0 disk.rbd0.out‚ 0.783203125 disk.dm-0.in‚ 68.61328125 disk.dm-0.out‚ 36201.1328125# proc.memory.rssMB‚ 3013.89453125 disk.sdb1.in‚0 network.recv‚ 1.026061e+06 disk./.usageGB‚ 2995.4375# disk.nvme0n1.out‚18635.41796875 disk.nvme1n1.in‚ 9.62109375 disk.md0.out‚ 319.34375 disk.sdb2.in‚ 0.26953125 disk.sda.in‚1.671875 disk.sda2.out‚332.2158203125 disk.nvme0n1.in‚ 58.9921875 disk.md0.in‚ 1.94140625 disk.md0p1.in‚ 1.94140625* proc.memory.percent‚18.395352363586426 disk.sdb2.out‚332.2158203125 disk.sda.out‚332.2158203125( proc.memory.availableMB‚ 1318.8828125 disk.sdb1.out‚0) disk./.usagePercent‚83.75368618799013 disk.sdb.in‚ 0.26953125 disk.rbd2.out‚ 0.845703125 disk.sda1.out‚0 disk.rbd0.in‚0 network.sent‚ 5.572192e+06# memory_percent‚91.9501781463623úP‘‹‚(:† «¨éÔ¢®îl gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚128.388' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.097 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö48oŒ‚(:ä «¨éÔÀëým network.recv‚ 1.03125e+06% disk./.usageGB‚2995.9374961853027 disk.dm-0.out‚36435.96484375 disk.sdb.out‚334.13623046875 disk.md0.in‚ 1.94140625# proc.memory.rssMB‚ 3013.89453125 network.sent‚ 5.678687e+06 disk.md0p1.out‚ 321.19921875 disk.rbd2.out‚ 0.845703125 disk.sdb1.in‚0 disk.sdb.in‚ 0.26953125 disk.nvme0n1.in‚ 58.9921875* proc.memory.percent‚18.395352363586426 cpu‚25.163821795309605 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.dm-0.in‚ 68.61328125 disk.sda2.in‚1.671875! disk.nvme0n1.out‚ 18870.171875 disk.sda1.out‚0) disk./.usagePercent‚83.76766629059586 disk.rbd0.in‚0 disk.rbd0.out‚ 0.783203125! disk.sda2.out‚334.13623046875# disk.nvme1n1.out‚17565.79296875 disk.sda.out‚334.13623046875$ memory_percent‚91.95027351379395 disk.nvme1n1.in‚ 9.62109375 disk.sdb2.in‚ 0.26953125 disk.sda.in‚1.671875 disk.sda1.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚ 1.94140625 disk.md0.out‚ 321.19921875! disk.sdb2.out‚334.13623046875( proc.memory.availableMB‚ 1318.8671875 ›Q`e‚(:Ú º¨éÔÍ»‹m) disk./.usagePercent‚83.76766629059586 disk.sda.out‚ 335.466796875 disk.sda1.in‚0 disk.sdb1.in‚0# memory_percent‚95.9679365158081* proc.memory.percent‚18.395352363586426 disk.rbd0.in‚0 disk.sda2.out‚ 335.466796875 disk.rbd2.out‚ 0.845703125 disk.sda.in‚1.671875 disk.md0p1.out‚ 322.45703125 disk.dm-0.in‚ 68.61328125 disk.sda2.in‚1.671875# disk.nvme1n1.out‚17566.16015625 disk.md0.in‚ 1.94140625 disk.md0.out‚ 322.45703125 network.sent‚ 5.68716e+06 disk.sdb2.in‚ 0.26953125 disk.sdb2.out‚ 335.466796875 disk.sda1.out‚0 disk.nvme0n1.in‚ 58.9921875# proc.memory.rssMB‚ 3013.89453125% disk./.usageGB‚2995.9374961853027 disk.rbd0.out‚ 0.783203125 disk.sdb1.out‚0 disk.md0p1.in‚ 1.94140625 disk.sdb.in‚ 0.26953125 disk.sdb.out‚ 335.466796875 proc.cpu.threads‚17 network.recv‚ 1.036245e+06 disk.dm-0.out‚36574.05078125 disk.nvme1n1.in‚ 9.62109375 disk.rbd2.in‚0( proc.memory.availableMB‚ 660.61328125 cpu‚25.16239234847534! disk.nvme0n1.out‚ 19007.890625ø ÈГŽ‚(:ˆ º¨éÔ‚ľn gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚119.091' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.77275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EÈÙ’‚(:‡ ɨéÔ§’Òm gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚117.525' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.38125 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t“Ø£n‚(:ã ɨéÔù±ën! disk.sdb2.out‚337.14599609375 disk.sdb.out‚337.14599609375# disk.nvme0n1.out‚19260.83203125 disk.sdb2.in‚ 0.26953125 disk.sda1.out‚0 disk.rbd0.out‚ 0.783203125 disk.sdb1.in‚0 network.recv‚ 1.039768e+06 disk.sda.in‚1.671875 disk.sdb1.out‚0 disk.md0p1.in‚ 1.94140625! disk.sda2.out‚337.14599609375# disk.nvme1n1.out‚17566.26953125 disk.rbd2.in‚0 cpu‚25.16165786959078 network.sent‚ 5.692452e+06 disk.md0p1.out‚ 324.0703125 disk.dm-0.out‚ 36827.1015625 disk.sda2.in‚1.671875 disk.nvme0n1.in‚ 58.9921875 disk.md0.out‚ 324.0703125 proc.cpu.threads‚17) disk./.usagePercent‚83.76766629059586 disk.sda.out‚337.14599609375 disk.nvme1n1.in‚ 9.62109375 disk.md0.in‚ 1.94140625$ memory_percent‚95.96607685089111* proc.memory.percent‚18.395352363586426% disk./.usageGB‚2995.9374961853027 disk.sda1.in‚0 disk.dm-0.in‚ 68.61328125 disk.sdb.in‚ 0.26953125# proc.memory.rssMB‚ 3013.89453125 disk.rbd0.in‚0 disk.rbd2.out‚ 0.845703125( proc.memory.availableMB‚ 660.91796875¬6/l2‘- _wandbruntime‚3076  _runtime‚3076ßEÉ-Å’¿ % _timestamp‚1.7884990183913217e+09 _runtime‚3076.168506144 _step‚214 % lr-AdamW/pg1‚0.001007528361262342 % lr-AdamW/pg2‚0.001007528361262342 trainer/global_step‚2649Á¬wªÊ“Ä trainer/global_step‚2649 % _timestamp‚1.7884990183913217e+09 _runtime‚3076.168506144 _step‚214 % lr-AdamW/pg1‚0.001007528361262342 % lr-AdamW/pg2‚0.001007528361262342Ö0¯„³2”- _wandbruntime‚3076  _runtime‚3076ݲL{ •ëO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55384770 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3023584170663 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _step‚215 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚226186240309760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67083735 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚246208819002880 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3022676195663 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚215772149410608 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚249382632973312 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 trainer/global_step‚2649 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚615 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976896 UADeviceStatsMonitoÒ¡xnvr.on_train_batch_start/requested_bytes.all.freed‚218794825606271 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚226185312335872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49320872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 $ _timestamp‚1.788499018392358e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚226186240309760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33184110 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚617 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚246208842942976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚215772166449968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122494848 _runtime‚3076.169682359 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122468503 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55384770 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚223151290515968 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3034931790848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3173813970432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚218795750620631 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640ï¸>Kö'–ðO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚215772166449968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122494848 _runtime‚3076.169682359 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122468503 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55384770 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚223151290515968 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3034931790848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3173813970432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚218795750620631 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55384770 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3023584170663 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _step‚215 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚226186240309760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67083735 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚246208819002880 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3022676195663 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚215772149410608 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚249382632973312 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 trainer/global_step‚2649 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚615 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976896 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚218794825606271 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚226185312335872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49320872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 $ _timestamp‚1.788499018392358e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚226186240309760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33184110 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚617 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚246208842942976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360×”Ó}ë2—- _wandbruntime‚3077  _runtime‚3077F3nü&˜öM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚223232927604736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚249474276545536 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚226268980869120 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3036053264384 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49339303 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67134935 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16142509 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚218875885488661 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122539865 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67108566 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963584 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚223232909601792 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 % _timestamp‚1.7884990195005379e+09 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49338684 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚218874960474093 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122513494 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55404928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3175207898112 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122539865 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3174974934528 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚223232927604736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16143126 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚226268052881920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚249474019641856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55404930 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122539865 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3174974934528 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚223232927604736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16143126 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚226268052881920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚249474019641856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55404930 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16142509 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚218875885488661 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122539865 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67108566 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963584 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚223232909601792 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 % _timestamp‚1.7884990195005379e+09 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49338684 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚218874960474093 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122513494 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55404928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3175207898112 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚230322354052096 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50213175 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚253952239874560 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚230323282025984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚219723046177168 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3091019215872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 241365504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3078553450771 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68354980 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚227232244807168 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68328637 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884990744909751e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124747750 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3090109244928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚230323282025984 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68328637 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16426450 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3232377014784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _step‚219 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚227232262810112 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 trainer/global_step‚2699 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚230322354052096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚250720104225280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚222801582588579>Òû$¨ñO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68354980 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚222802507602939 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56392770 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚250720128165376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3090109244928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚253952505180160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚227232244807168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33787360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚219723029137808 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56392770 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚219723046177168 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3091019215872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 241365504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3078553450771 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68354980 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚227232244807168 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68328637 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884990744909751e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124747750 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3090109244928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚230323282025984 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68328637 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16426450 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3232377014784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _step‚219 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚227232262810112 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 trainer/global_step‚2699 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚230322354052096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚250720104225280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚222801582588579 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚230322354052096 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚‡³ä50213175 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚253952239874560 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚230323282025984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚227313881895936 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56412928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68353468 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124792767 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56412930 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚219802046732352 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50230818 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚222882642470969 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3092140689408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3091230705152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚250810353869824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3233303854080 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3091230705152 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚3133.255558981 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚230406022585344 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚222881717456401 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3080578699257 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68379837 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚227313881895936 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _step‚220 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33799425 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3092140689408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚254043888590336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56412928 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68379837 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50231463 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230866432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124792767 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚227313863892992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16432036 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3233534720512 trainer/global_step‚2699 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 124766396 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68353468 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚250810329929728 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7884990754774618e+09x\•'«ûM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 124766396 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚230406022585344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚254043633783808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚230405094598144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚227313863892992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16431393 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56412930 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚227313881895936 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56412928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68353468 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124792767 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56412930 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚219802046732352 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50230818 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚222882642470969 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3092140689408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3091230705152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚250810353869824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3233303854080 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3091230705152 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚3133.255558981 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚230406022585344 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚222881717456401 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3080578699257 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68379837 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚227313881895936 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _step‚220 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33799425 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3092140689408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚254043888590336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56412928 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68379837 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50231463 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230866432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124792767 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚227313863892992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16432036 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3233534720512 trainer/global_step‚2699 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 124766396 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68353468 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚250810329929728 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7884990754774618e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0Ü0KM[2¬- _wandbruntime‚3133  _runtime‚3133'ô®ÏY­Ó $loss/total_loss‚45.7791737065194 1loss/SRqqt1/assignment_loss‚2.128469142175856 _runtime‚ 3133.2575707 /loss/FBt1/detection_loss‚0.6224768402044222 2loss/SRqqt2/assignment_loss‚2.0609835132956507 /loss/FRt4/detection_loss‚3.4964777419441626 0loss/SRqqt1/detection_loss‚2.219055408523196 epoch‚2 trainer/global_step‚2699 % _timestamp‚1.7884990754805942e+09 0loss/FRt1/assignment_loss‚1.9731158126484265 1loss/FBt1/assignment_loss‚0.21560098109535847 1loss/FBt2/assignment_loss‚0.35374149339986616 1loss/FBt3/assignment_loss‚0.32591276690696497 0loss/SRqqt3/detection_loss‚1.023576106745423 /loss/FBt3/detection_loss‚3.3912663885525296 /loss/FRt3/assignment_loss‚1.568601086284175 /loss/FRt3/detection_loss‚1.4972902434793385 /loss/FBt4/detection_loss‚2.9329730212688445 _step‚221 /loss/FRt2/detection_loss‚2.2908020300685235 /loss/FRt2/assignment_loss‚3.394603412106352 /loss/FRt4/assignment_loss‚3.543763411672492 1loss/FBt4/assignment_loss‚0.33929913269821554 /loss/FRt1/detection_loss‚1.7873864279985852 1loss/SRqqt2/detection_loss‚2.5812236964702606 1loss/SRqqt4/detection_loss‚2.4681368367425325 /loss/FBt2/detection_loss‚2.5689866772064796 1loss/SRqqt3/assignment_loss‚0.997921101834723 2loss/SRqqt4/assignment_loss‚1.9975104331970215y1/©^®Ø 2loss/SRqqt2/assignment_loss‚2.0609835132956507 /loss/FRt4/detection_loss‚3.4964777419441626 0loss/SRqqt1/detection_loss‚2.219055408523196 epoch‚2 trainer/global_step‚2699 % _timestamp‚1.7884990754805942e+09 0loss/FRt1/assignment_loss‚1.9731158126484265 1loss/FBt1/assignment_loss‚0.21560098109535847 1loss/FBt2/assignment_loss‚0.35374149339986616 1loss/FBt3/assignment_loss‚0.32591276690696497 0loss/SRqqt3/detection_loss‚1.023576106745423 /loss/FBt3/detection_loss‚3.3912663885525296 /loss/FRt3/assignment_loss‚1.568601086284175 /loss/FRt3/detection_loss‚1.4972902434793385 /loss/FBt4/detection_loss‚2.9329730212688445 _step‚221 /loss/FRt2/detection_loss‚2.2908020300685235 /loss/FRt2/assignment_loss‚3.394603412106352 /loss/FRt4/assignment_loss‚3.543763411672492 1loss/FBt4/assignment_loss‚0.33929913269821554 /loss/FRt1/detection_loss‚1.7873864279985852 1loss/SRqqt2/detection_loss‚2.5812236964702606 1loss/SRqqt4/detection_loss‚2.4681368367425325 /loss/FBt2/detection_loss‚2.5689866772064796 1loss/SRqqt3/assignment_loss‚0.997921101834723 2loss/SRqqt4/assignment_loss‚1.9975104331970215 $loss/total_loss‚45.7791737065194 1loss/SRqqt1/assignment_loss‚2.128469142175856 _runtime‚ 3133.2575707 /loss/FBt1/detection_loss‚0.6224768402044222Ýef <‘¯‚(:† …©éÔÍ›øm gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚169.61' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ߟg°‚(:Ü …©éÔöÙ±n# disk.nvme1n1.out‚17683.34765625 disk.sda.in‚1.671875 disk.sda.out‚343.8466796875 disk.sdb1.out‚0 disk.sda2.out‚343.8466796875 disk.rbd2.in‚0 disk.sdb2.in‚ 0.26953125 network.recv‚ 1.057515e+06) disk./.usagePercent‚83.74169637091379 disk.nvme0n1.in‚ 58.9921875 disk.nvme1n1.in‚ 9.62109375 disk.md0.in‚ 1.94140625* proc.memory.percent‚18.395352363586426 disk.rbd0.out‚ 0.783203125 disk.md0p1.out‚ 330.4921875 disk.rbd2.out‚ 0.845703125 network.sent‚ 5.814951e+06 disk.sdb1.in‚0 disk.dm-0.out‚37813.56640625 disk.md0.out‚ 330.4921875 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.in‚1.671875$ disk./.usageGB‚2995.008686065674$ memory_percent‚95.96724510192871# proc.memory.rssMB‚ 3013.89453125 cpu‚25.158279571924687 disk.sdb.out‚343.8466796875 disk.nvme0n1.out‚ 20130.21875 disk.sdb2.out‚343.8466796875' proc.memory.availableMB‚ 660.7265625 disk.sda1.out‚0 disk.rbd0.in‚0 disk.md0p1.in‚ 1.94140625 disk.dm-0.in‚ 68.61328125 disk.sdb.in‚ 0.26953125ç2x*“±‚(:ˆ ”©éÔ§ò½k gpu.0.gpu‚55.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚180.805' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.20125 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×VÞÚn²‚(:ã ”©éÔŸÚ¦o' proc.memory.availableMB‚ 660.6953125 network.sent‚ 5.922379e+06) disk./.usagePercent‚83.73821113136619 disk.sda2.in‚1.671875 disk.rbd2.in‚0 disk.sda.in‚1.671875 disk.md0p1.out‚ 332.3984375# proc.memory.rssMB‚ 3013.89453125% disk./.usageGB‚2994.8840370178223 disk.dm-0.in‚ 68.61328125 disk.dm-0.out‚37891.43359375 disk.sda1.in‚0 disk.nvme0n1.in‚ 58.9921875 disk.md0.in‚ 1.94140625$ memory_percent‚95.96743583679199* proc.memory.percent‚18.395352363586426 proc.cpu.threads‚17# disk.nvme0n1.out‚20207.94140625 disk.sdb2.in‚ 0.26953125 disk.md0p1.in‚ 1.94140625 network.recv‚ 1.063157e+06 disk.sdb.in‚ 0.26953125! disk.sda2.out‚345.82861328125 disk.sda.out‚345.82861328125 disk.rbd0.out‚ 0.783203125 disk.sdb.out‚345.82861328125! disk.sdb2.out‚345.82861328125 disk.sdb1.out‚0 cpu‚25.157516560918655 disk.nvme1n1.in‚ 9.62109375" disk.nvme1n1.out‚ 17683.4921875 disk.md0.out‚ 332.3984375 disk.rbd2.out‚ 0.845703125 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sdb1.in‚0XkéY“³‚(:ˆ £©éÔâºËm gpu.0.gpu‚51.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.129' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.03225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%zâ8n´‚(:ã £©éÔòèæm disk.sda1.in‚0 disk.sda2.in‚1.671875 disk.nvme1n1.in‚ 9.62109375 disk.rbd2.in‚0! disk.sdb2.out‚347.49072265625 disk.md0.out‚ 333.99609375 disk.sdb2.in‚ 0.26953125 disk.sda1.out‚0$ memory_percent‚95.96586227416992* proc.memory.percent‚18.395352363586426 cpu‚25.15672270869815% disk./.usageGB‚2994.2660331726074 disk.sdb1.out‚0 disk.md0p1.out‚ 333.99609375 disk.nvme0n1.in‚ 58.9921875 disk.sda.in‚1.671875 disk.sdb1.in‚0 network.recv‚ 1.067371e+06 disk.dm-0.out‚ 37944.3828125# disk.nvme0n1.out‚20260.66015625 disk.rbd0.out‚ 0.783203125 proc.cpu.threads‚17 network.sent‚ 5.928211e+06! disk.sda2.out‚347.49072265625 disk.rbd2.out‚ 0.845703125 disk.sda.out‚347.49072265625 disk.md0p1.in‚ 1.94140625) disk./.usagePercent‚83.72093148519927 disk.dm-0.in‚ 68.61328125 disk.sdb.in‚ 0.26953125 disk.sdb.out‚347.49072265625 disk.rbd0.in‚0& proc.memory.availableMB‚ 660.953125# proc.memory.rssMB‚ 3013.89453125# disk.nvme1n1.out‚17683.72265625 disk.md0.in‚ 1.94140625¢áÕÈ’µ‚(:‡ ²©éÔ±ÕÅk gpu.0.gpu‚51.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚176.738' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.1845 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹û+qc¶‚(:Ø ²©éÔ¤¨ºn disk.rbd2.out‚ 0.845703125 disk.sda.out‚ 348.947265625 cpu‚25.15605637691949( disk./.usagePercent‚83.7248620324685 disk.rbd2.in‚0 disk.sda.in‚1.671875 disk.rbd0.out‚ 0.783203125 disk.sdb1.in‚0 disk.md0p1.in‚ 1.94140625 disk.nvme1n1.in‚ 9.62109375 disk.md0.out‚ 335.39453125 disk.sdb2.in‚ 0.26953125 disk.sda2.in‚1.671875( proc.memory.availableMB‚ 660.58984375 proc.cpu.threads‚17 disk.dm-0.in‚ 68.61328125 disk.nvme0n1.in‚ 58.9921875" disk.nvme0n1.out‚ 20304.9453125! disk.nvme1n1.out‚ 17756.265625 disk.sdb2.out‚ 348.947265625 disk.rbd0.in‚0 disk.sda2.out‚ 348.947265625* proc.memory.percent‚18.395352363586426 network.sent‚ 5.936693e+06 network.recv‚ 1.072068e+06 disk.dm-0.out‚ 38061.2109375 disk.sdb.out‚ 348.947265625$ memory_percent‚95.96807956695557$ disk./.usageGB‚2994.406608581543 disk.sda1.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 3013.89453125 disk.sdb1.out‚0 disk.md0p1.out‚ 335.39453125 disk.sdb.in‚ 0.26953125 disk.md0.in‚ 1.94140625™çÔø2·- _wandbruntime‚3185  _runtime‚3185ÓـǸÁ & lr-AdamW/pg1‚0.0010011927916732282 & lr-AdamW/pg2‚0.0010011927916732282 trainer/global_step‚2749 % _timestamp‚1.7884991280961466e+09 _runtime‚3185.873498691 _step‚222ŸH ?Ì¹Æ & lr-AdamW/pg1‚0.0010011927916732282 & lr-AdamW/pg2‚0.0010011927916732282 trainer/global_step‚2749 % _timestamp‚1.7884991280961466e+09 _runtime‚3185.873498691 _step‚222Þ˜†2º- _wandbruntime‚3185  _runtime‚3185r{Ç’»ìO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69573539 trainer/global_step‚2749 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57400768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3290565114880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚226809264585247 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚234459395768320 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚620 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226685440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127000652 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚622 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3146196669952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51104014 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 RBDeviceStatsMonitor.on_train_batch_start=RÁÒ`/reserved_bytes.all.current‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚231313217101312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34390610 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69573539 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3135338680879 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69599882 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚234460323742208 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51104636 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3147106640896 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 _step‚223 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126974307 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16714024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚234459395768320 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚258522205188096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34390612 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3134430705879 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57400770 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚231313199098368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69599882 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚255231389447680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16713404 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 % _timestamp‚1.7884991280971093e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57400768 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚223673908865008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚226808339570887 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚231313217101312 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚234460323742208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3146196669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚258521954562560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚255231413387776 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625536 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856îáýí÷'¼ñO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69599882 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚234460323742208 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51104636 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3147106640896 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 _step‚223 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126974307 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16714024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚234459395768320 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚258522205188096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34390612 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3134430705879 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57400770 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚231313199098368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69599882 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚255231389447680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16713404 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 % _timestamp‚1.7884991280971093e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57400768 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚223673908865008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚226808339570887 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚231313217101312 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚234460323742208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3146196669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚258521954562560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚255231413387776 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625536 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69573539 trainer/global_step‚2749 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57400768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3290565114880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚226809264585247 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚234459395768320 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚620 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226685440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127000652 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚622 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3146196669952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51104014 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚231313217101312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34390610 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69573539 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3135338680879 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144ßÎRª2½- _wandbruntime‚3186  _runtime‚31866¶¾ü&¾öM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34402675 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚226888474438709 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127045669 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16719099 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57420930 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51122386 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚234543064301568 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚226889399453277 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3135547979157 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16719709 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚3186.916037726 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚258613363047424 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚255321615152128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3291966178816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57420928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3148228114432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3147318130176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127019298 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51121774 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69598370 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚612 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚223752943498912 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3147318130176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚255321639092224 % _timestamp‚1.7884991291378527e+09 _step‚224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚234542136314368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚231394818184192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚231394818184192 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57420928zÝÛX¿ûM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34402675 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚226888474438709 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127045669 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16719099 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57420930 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51122386 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚234543064301568 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚226889399453277 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3135547979157 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16719709 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚3186.916037726 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚258613363047424 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚255321615152128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3291966178816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57420928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3148228114432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3147318130176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127019298 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51121774 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69598370 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚612 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚223752943498912 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3147318130176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚255321639092224 % _timestamp‚1.7884991291378527e+09 _step‚224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚234542136314368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚231394818184192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚231394818184192 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57420928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570àã#é2À- _wandbruntime‚3186  _runtime‚3186ÖÿJWÁÑ /loss/FBt3/detection_loss‚2.5218403975168866 _step‚225 2loss/SRqqt1/assignment_loss‚1.3421840198676696 0loss/FBt3/assignment_loss‚0.3108111855923198 .loss/FRt4/detection_loss‚3.305344993417913 %loss/total_loss‚45.01123043677856 0loss/FRt3/assignment_loss‚1.4913231106474996 .loss/FRt1/detection_loss‚2.214127231881572 1loss/SRqqt3/detection_loss‚1.0750257324596697 0loss/SRqqt4/detection_loss‚3.091693812608719 1loss/SRqqt3/assignment_loss‚1.200057811831886 0loss/SRqqt1/detection_loss‚2.570852742953734 /loss/FBt1/detection_loss‚0.9248591229319573 .loss/FRt3/detection_loss‚1.688269944046624 epoch‚2 trainer/global_step‚2749 _runtime‚3186.917237532 2loss/SRqqt2/assignment_loss‚1.7599424502501886 2loss/SRqqt4/assignment_loss‚1.6294459303840996 .loss/FBt4/detection_loss‚2.412355876631207 % _timestamp‚1.7884991291401656e+09 .loss/FRt2/assignment_loss‚3.07715415648925 1loss/FBt1/assignment_loss‚0.28474865168404934 /loss/FBt2/detection_loss‚2.4564795911312105 0loss/FRt1/assignment_loss‚3.1652893931307693 0loss/FRt4/assignment_loss‚2.9253451607444068 0loss/SRqqt2/detection_loss‚2.655366244415442 1loss/FBt2/assignment_loss‚0.18240999285790166 1loss/FBt4/assignment_loss‚0.11955187233545278 .loss/FRt2/detection_loss‚2.606751010968135iÏž\ÂÖ .loss/FRt4/detection_loss‚3.305344993417913 %loss/total_loss‚45.01123043677856 0loss/FRt3/assignment_loss‚1.4913231106474996 .loss/FRt1/detection_loss‚2.214127231881572 1loss/SRqqt3/detection_loss‚1.0750257324596697 0loss/SRqqt4/detection_loss‚3.091693812608719 1loss/SRqqt3/assignment_loss‚1.200057811831886 0loss/SRqqt1/detection_loss‚2.570852742953734 /loss/FBt1/detection_loss‚0.9248591229319573 .loss/FRt3/detection_loss‚1.688269944046624 epoch‚2 trainer/global_step‚2749 _runtime‚3186.917237532 2loss/SRqqt2/assignment_loss‚1.7599424502501886 2loss/SRqqt4/assignment_loss‚1.6294459303840996 .loss/FBt4/detection_loss‚2.412355876631207 % _timestamp‚1.7884991291401656e+09 .loss/FRt2/assignment_loss‚3.07715415648925 1loss/FBt1/assignment_loss‚0.28474865168404934 /loss/FBt2/detection_loss‚2.4564795911312105 0loss/FRt1/assignment_loss‚3.1652893931307693 0loss/FRt4/assignment_loss‚2.9253451607444068 0loss/SRqqt2/detection_loss‚2.655366244415442 1loss/FBt2/assignment_loss‚0.18240999285790166 1loss/FBt4/assignment_loss‚0.11955187233545278 .loss/FRt2/detection_loss‚2.606751010968135 /loss/FBt3/detection_loss‚2.5218403975168866 _step‚225 2loss/SRqqt1/assignment_loss‚1.3421840198676696 0loss/FBt3/assignment_loss‚0.3108111855923198á³Djk“Â(:ˆ Á©éÔ°ñ¨m gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.129' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.03225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01kú¹[Ä‚(:Ð Á©éÔà¿Œn disk.md0.out‚ 337.46875 network.sent‚ 5.93989e+06$ disk./.usageGB‚2994.436138153076 disk.nvme0n1.in‚ 58.9921875* proc.memory.percent‚18.395352363586426 proc.cpu.threads‚17 disk.rbd0.out‚ 0.783203125 disk.sdb1.out‚0# disk.nvme0n1.out‚20344.91796875 disk.md0.in‚ 1.94140625% proc.memory.availableMB‚ 660.59375 disk.sda.out‚ 351.087890625 disk.sda1.in‚0 disk.md0p1.out‚ 337.46875 disk.dm-0.out‚ 38169.9375 disk.sda2.in‚1.671875 disk.rbd2.in‚0 disk.nvme1n1.in‚ 9.62109375 cpu‚25.155180132322446) disk./.usagePercent‚83.72568769164765 disk.sda.in‚1.671875 disk.dm-0.in‚ 68.61328125 disk.sda2.out‚ 351.087890625 network.recv‚ 1.075801e+06 disk.sdb2.out‚ 351.087890625 disk.sdb1.in‚0 disk.md0p1.in‚ 1.94140625# disk.nvme1n1.out‚17828.89453125 disk.rbd0.in‚0 disk.sdb2.in‚ 0.26953125 disk.sda1.out‚0 disk.sdb.in‚ 0.26953125 disk.sdb.out‚ 351.087890625 disk.rbd2.out‚ 0.845703125$ memory_percent‚95.96805572509766# proc.memory.rssMB‚ 3013.89453125­ÐÌ “Å‚(:ˆ ЩéÔÔÜÿl gpu.0.gpu‚22.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚129.005' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.25125 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”y°\iÆ‚(:Þ ЩéÔб¹m% disk./.usageGB‚2994.5006561279297 disk.sdb1.out‚0 disk.sdb.in‚ 0.26953125 disk.sdb2.in‚ 0.26953125 cpu‚25.15447114776222 network.sent‚ 6.04946e+06 disk.nvme1n1.in‚ 9.62109375( proc.memory.availableMB‚ 660.83984375 disk.sdb.out‚352.5888671875 disk.nvme0n1.in‚ 58.9921875 disk.sda.out‚352.5888671875 proc.cpu.threads‚17 network.recv‚ 1.082009e+06) disk./.usagePercent‚83.72749164122742 disk.rbd0.in‚0 disk.md0p1.in‚ 1.94140625 disk.md0.out‚ 338.8984375 disk.rbd2.out‚ 0.845703125 disk.sda.in‚1.671875 disk.sda1.in‚0 disk.sda2.in‚1.671875" disk.nvme1n1.out‚ 17885.8671875 disk.rbd2.in‚0* proc.memory.percent‚18.395352363586426 disk.md0p1.out‚ 338.8984375 disk.sda1.out‚0 disk.rbd0.out‚ 0.783203125 disk.sdb1.in‚0# disk.nvme0n1.out‚20379.33984375 disk.dm-0.in‚ 68.61328125 disk.sdb2.out‚352.5888671875 disk.dm-0.out‚38265.20703125 disk.sda2.out‚352.5888671875 disk.md0.in‚ 1.94140625$ memory_percent‚95.96655368804932# proc.memory.rssMB‚ 3013.894531257Z$y“Ç‚(:ˆ ß©éÔ‹©·k gpu.0.gpu‚52.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚122.153' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.53825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö¬…jÈ‚(:ß ß©éÔä¾²l# disk.nvme1n1.out‚18043.23046875 disk.md0p1.in‚ 1.94140625 disk.dm-0.in‚ 68.61328125 disk.dm-0.out‚38500.16796875$ memory_percent‚95.96807956695557 network.sent‚ 6.053061e+06 network.recv‚ 1.085998e+06 disk.nvme0n1.in‚ 58.9921875 disk.md0.out‚ 340.390625# proc.memory.rssMB‚ 3013.89453125! disk.sda2.out‚354.14013671875 disk.md0.in‚ 1.94140625 disk.rbd2.in‚0 cpu‚25.153852187335584 proc.cpu.threads‚17 disk.sdb2.in‚ 0.26953125 disk.sda.in‚1.671875 disk.sda.out‚354.14013671875 disk.sda1.out‚0 disk.sdb.out‚354.14013671875 disk.sda2.in‚1.671875 disk.nvme1n1.in‚ 9.62109375 disk.rbd2.out‚ 0.845703125 disk.rbd0.out‚ 0.783203125 disk.md0p1.out‚ 340.390625* proc.memory.percent‚18.395352363586426! disk.sdb2.out‚354.14013671875 disk.sdb1.in‚0 disk.sdb1.out‚0( proc.memory.availableMB‚ 660.58984375) disk./.usagePercent‚83.73622756545032% disk./.usageGB‚2994.8130950927734 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sdb.in‚ 0.26953125 disk.nvme0n1.out‚ 20456.9375m‰Xˆ’É‚(:‡ î©éÔÚéÑm gpu.0.gpu‚12.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.266' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.3165 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rË ]Ê‚(:Ò î©éÔ‰Óßn disk.sda1.out‚0 disk.sda2.in‚1.671875 disk.nvme0n1.in‚ 58.9921875 disk.rbd2.out‚ 0.845703125 cpu‚25.153260767187387# disk.nvme0n1.out‚20521.36328125 disk.sda.in‚1.671875* proc.memory.percent‚18.395352363586426 disk.md0.in‚ 1.94140625 network.sent‚ 6.061582e+06 disk.sda1.in‚0 disk.sdb.in‚ 0.26953125# disk.nvme1n1.out‚18123.80078125 disk.md0.out‚ 341.51953125# proc.memory.rssMB‚ 3013.89453125 network.recv‚ 1.090899e+06% disk./.usagePercent‚ 83.7467127224 disk.rbd0.out‚ 0.783203125 disk.sda2.out‚ 355.32421875# memory_percent‚95.9665298461914% disk./.usageGB‚2995.1880950927734 disk.sdb1.out‚0 disk.md0p1.in‚ 1.94140625 disk.sdb.out‚ 355.32421875 disk.nvme1n1.in‚ 9.62109375 disk.sdb2.in‚ 0.26953125 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.dm-0.out‚ 38645.1640625 disk.sdb2.out‚ 355.32421875 disk.sda.out‚ 355.32421875% proc.memory.availableMB‚ 660.84375 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.md0p1.out‚ 341.51953125 disk.dm-0.in‚ 68.61328125a[2Ë- _wandbruntime‚3241  _runtime‚3241ß…ÇÌÁ _runtime‚3241.901348963 _step‚226 & lr-AdamW/pg1‚0.0009978761826955757 & lr-AdamW/pg2‚0.0009978761826955757 trainer/global_step‚2799 % _timestamp‚1.7884991841241152e+09þ+ìmÌÍÆ & lr-AdamW/pg1‚0.0009978761826955757 & lr-AdamW/pg2‚0.0009978761826955757 trainer/global_step‚2799 % _timestamp‚1.7884991841241152e+09 _runtime‚3241.901348963 _step‚226âý¶B2Î- _wandbruntime‚3241  _runtime‚3241”Sß_ò'ÏìO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚3241.902427018 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3202284094976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3203194065920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚227624788592208 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58408768 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚259742698610176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879744 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70844784 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70844784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3191215935987 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17002418 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚259742674670080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚238596437484544 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238596437484544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129253554 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3348929477120 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58408770 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70818441 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚238597365458432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51995657 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34993862 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚235394153389568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3190307960987 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34993860 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚230816021567555 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚230815096553195 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58408768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17001797 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚227624805631568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70818441 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚263091858967040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚235394171392512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0˜G[/5ÐñO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 129227209 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3202284094976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58408770 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚263091604147200 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 _step‚227 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51996280 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚235394171392512 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70844784 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70844784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3191215935987 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17002418 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚259742674670080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚238596437484544 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238596437484544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129253554 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3348929477120 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58408770 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70818441 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚238597365458432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51995657 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34993862 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚235394153389568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3190307960987 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34993860 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚230816021567555 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚230815096553195 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58408768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17001797 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚227624805631568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70818441 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚263091858967040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚235394171392512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_s¹r©ø tart/allocation.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚3241.902427018 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3202284094976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3203194065920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚227624788592208 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58408768 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚259742698610176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879744 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚228 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70843272 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35005927 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚227703806186752 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚238680106017792 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3191425234265 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129298571 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58428930 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚2799 $ _timestamp‚1.788499185313736e+09 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70869641 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129272200 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚238679178030592 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚259832924314624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _runtime‚3243.091629034 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70869641 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚235475790478336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚259832900374528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3350106736128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚230896156435585 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 129272200 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129298571 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52013376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚235475772475392 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17007451 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570à$—Ã'ÓúM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚227703806186752 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚238680106017792 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3191425234265 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129298571 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58428930 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚2799 $ _timestamp‚1.788499185313736e+09 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70869641 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129272200 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚238679178030592 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚259832924314624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _runtime‚3243.091629034 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70869641 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚235475790478336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚259832900374528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3350106736128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚230896156435585 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 129272200 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129298571 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52013376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚235475772475392 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17007451 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚228 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70843272 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35005927 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856äY<ïš2Ô- _wandbruntime‚3243  _runtime‚3243ƒLŽ½]Õ× 2loss/SRqqt2/assignment_loss‚1.5306189894676208 1loss/SRqqt4/detection_loss‚2.9809634298086167 /loss/FBt4/detection_loss‚1.7458999225726495 epoch‚2 0loss/FRt3/assignment_loss‚1.7063782569290935 .loss/FRt4/detection_loss‚3.443078244725863 /loss/FBt1/detection_loss‚0.9032829865127018 /loss/FRt1/assignment_loss‚2.269154023616872 2loss/SRqqt1/assignment_loss‚1.3989326700016305 2loss/SRqqt4/assignment_loss‚1.1639410821924685 2loss/FBt3/assignment_loss‚0.004866975941695273 0loss/SRqqt2/detection_loss‚2.808057153224945 /loss/FRt1/detection_loss‚1.8475748915723749 % _timestamp‚1.7884991853163774e+09 2loss/SRqqt3/assignment_loss‚1.0056907118508411 0loss/FBt2/assignment_loss‚0.2531657854073833 trainer/global_step‚2799 0loss/FRt4/assignment_loss‚3.8198816776275635 /loss/FBt4/assignment_loss‚0.559260968522197 /loss/FRt2/detection_loss‚2.8242585878161823 /loss/FRt3/detection_loss‚1.5856304106073102 1loss/SRqqt3/detection_loss‚0.9352970183977196 .loss/FBt2/detection_loss‚2.489164826961664 _runtime‚3243.093429563 _step‚229 0loss/FRt2/assignment_loss‚3.7840221208684586 0loss/SRqqt1/detection_loss‚2.341463175084856 /loss/FBt3/detection_loss‚5.7280289232730865 &loss/total_loss‚47.475164772317356 1loss/FBt1/assignment_loss‚0.34655193933356554ÐA bÖÜ 0loss/FRt3/assignment_loss‚1.7063782569290935 .loss/FRt4/detection_loss‚3.443078244725863 /loss/FBt1/detection_loss‚0.9032829865127018 /loss/FRt1/assignment_loss‚2.269154023616872 2loss/SRqqt1/assignment_loss‚1.3989326700016305 2loss/SRqqt4/assignment_loss‚1.1639410821924685 2loss/FBt3/assignment_loss‚0.004866975941695273 0loss/SRqqt2/detection_loss‚2.808057153224945 /loss/FRt1/detection_loss‚1.8475748915723749 % _timestamp‚1.7884991853163774e+09 2loss/SRqqt3/assignment_loss‚1.0056907118508411 0loss/FBt2/assignment_loss‚0.2531657854073833 trainer/global_step‚2799 0loss/FRt4/assignment_loss‚3.8198816776275635 /loss/FBt4/assignment_loss‚0.559260968522197 /loss/FRt2/detection_loss‚2.8242585878161823 /loss/FRt3/detection_loss‚1.5856304106073102 1loss/SRqqt3/detection_loss‚0.9352970183977196 .loss/FBt2/detection_loss‚2.489164826961664 _runtime‚3243.093429563 _step‚229 0loss/FRt2/assignment_loss‚3.7840221208684586 0loss/SRqqt1/detection_loss‚2.341463175084856 /loss/FBt3/detection_loss‚5.7280289232730865 &loss/total_loss‚47.475164772317356 1loss/FBt1/assignment_loss‚0.34655193933356554 2loss/SRqqt2/assignment_loss‚1.5306189894676208 1loss/SRqqt4/detection_loss‚2.9809634298086167 /loss/FBt4/detection_loss‚1.7458999225726495 epoch‚2å9{.‘“ׂ(:ˆ ý©éÔÓËÈm gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚174.903' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.72575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00º&cØ‚(:Ø ý©éÔðí¦n disk.md0.in‚ 1.94140625 disk.sda.out‚357.4423828125 disk.sda1.in‚0 disk.sdb.in‚ 0.26953125 disk.sda.in‚1.671875 disk.rbd0.in‚0 network.sent‚ 6.064737e+06 disk.sdb2.out‚357.4423828125 disk.dm-0.in‚ 68.61328125 proc.cpu.threads‚17 disk.nvme1n1.out‚ 18201.125 disk.md0p1.in‚ 1.94140625' proc.memory.availableMB‚ 660.8359375# proc.memory.rssMB‚ 3013.89453125 cpu‚25.15256701651184 disk.rbd2.out‚ 0.845703125 disk.sdb2.in‚ 0.26953125 disk.sda1.out‚0 disk.rbd0.out‚ 0.783203125 disk.md0p1.out‚ 343.5703125) disk./.usagePercent‚83.76776463160647 disk.md0.out‚ 343.5703125 disk.sdb1.in‚0 disk.sdb.out‚357.4423828125$ memory_percent‚95.96657752990723 network.recv‚ 1.094509e+06 disk.sdb1.out‚0 disk.dm-0.out‚ 38813.4140625 disk.nvme0n1.in‚ 58.9921875" disk.nvme0n1.out‚ 20612.2890625* proc.memory.percent‚18.395352363586426% disk./.usageGB‚2995.9410133361816 disk.nvme1n1.in‚ 9.62109375 disk.rbd2.in‚0 disk.sda2.in‚1.671875 disk.sda2.out‚357.4423828125z½”Ó‘Ù‚(:† ŒªéÔÓúøl gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚121.876' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.469 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–|I£lÚ‚(:á ŒªéÔ”ùÊm disk.rbd0.in‚0 disk.rbd0.out‚ 0.783203125 disk.sdb1.in‚0 disk.rbd2.out‚ 0.845703125# proc.memory.rssMB‚ 3013.89453125 proc.cpu.threads‚17 disk.sda.in‚1.671875 disk.sdb.out‚359.23388671875* proc.memory.percent‚18.395352363586426 cpu‚25.151883987746345 disk.nvme0n1.in‚ 58.9921875" disk.nvme1n1.out‚ 18358.5234375 network.recv‚ 1.100683e+06 disk.dm-0.out‚39034.98828125 disk.md0.out‚ 345.296875 disk.rbd2.in‚0$ memory_percent‚95.96688747406006$ disk./.usageGB‚2995.938995361328! disk.sda2.out‚359.23388671875 network.sent‚ 6.174453e+06 disk.sdb1.out‚0 disk.md0p1.in‚ 1.94140625 disk.md0p1.out‚ 345.296875 disk.dm-0.in‚ 68.61328125# disk.nvme0n1.out‚20676.46484375 disk.nvme1n1.in‚ 9.62109375 disk.md0.in‚ 1.94140625) disk./.usagePercent‚83.76770820818497 disk.sdb2.in‚ 0.26953125 disk.sdb.in‚ 0.26953125 disk.sda2.in‚1.671875! disk.sdb2.out‚359.23388671875 disk.sda1.out‚0( proc.memory.availableMB‚ 660.78515625 disk.sda.out‚359.23388671875 disk.sda1.in‚0ªaÜÛ‚(:’ ›ªéÔܵ¡m gpu.0.gpu‚23.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚119.978' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.994499999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0о8ùdÜ‚(:Ù ›ªéÔ‚…¤n disk.sda1.in‚0 disk.rbd0.out‚ 0.783203125 disk.md0p1.in‚ 1.94140625 disk.sda2.in‚1.671875! disk.sda2.out‚360.59521484375$ disk./.usageGB‚2995.937976837158! disk.sdb2.out‚360.59521484375 disk.rbd2.in‚0$ memory_percent‚95.96681594848633 disk.sdb2.in‚ 0.26953125 disk.dm-0.out‚ 39362.28125 disk.md0.out‚ 346.609375 cpu‚25.15133577689043 network.sent‚ 6.177883e+06 disk.sdb1.in‚0" disk.nvme0n1.out‚ 20741.6171875 disk.md0.in‚ 1.94140625 disk.sda.out‚360.59521484375 disk.sda1.out‚0 disk.sdb1.out‚0" disk.nvme1n1.out‚ 18620.6640625# proc.memory.rssMB‚ 3013.89453125 proc.cpu.threads‚17 network.recv‚ 1.104239e+06 disk.sda.in‚1.671875 disk.md0p1.out‚ 346.609375 disk.sdb.in‚ 0.26953125 disk.rbd2.out‚ 0.845703125 disk.dm-0.in‚ 68.61328125 disk.sdb.out‚360.59521484375 disk.nvme0n1.in‚ 58.9921875 disk.nvme1n1.in‚ 9.62109375* proc.memory.percent‚18.395352363586426( disk./.usagePercent‚83.7676797298229 disk.rbd0.in‚0& proc.memory.availableMB‚ 660.796875ºÌ 2Ý- _wandbruntime‚3298  _runtime‚3298}xaÇÞÁ & lr-AdamW/pg2‚0.0009944612722876751 trainer/global_step‚2849 % _timestamp‚1.7884992410446403e+09 _runtime‚3298.822211299 _step‚230 & lr-AdamW/pg1‚0.0009944612722876751'¯KÌßÆ _step‚230 & lr-AdamW/pg1‚0.0009944612722876751 & lr-AdamW/pg2‚0.0009944612722876751 trainer/global_step‚2849 % _timestamp‚1.7884992410446403e+09 _runtime‚3298.822211299æþZr2à- _wandbruntime‚3298  _runtime‚3298iœ’Ç·áìO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72089686 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚264253959892480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚267661306641408 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚267661062307328 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚231575668319408 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 % _timestamp‚1.7884992410460181e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131480111 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59416768 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17290195 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚24273347920jnÒ; 0768 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131506456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3258371520000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3247093191095 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3259281490944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52887928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17290816 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚264253983832576 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚231575685358768 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35597110 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚242733479200768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3259281490944 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚234821853535503 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _step‚231 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59416768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚239475107680768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚242734407174656 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59416770 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72089686 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52887928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17290816 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚264253983832576 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚231575685358768 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35597110 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚242733479200768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3259281490944 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚234821853535503 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _step‚231 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59416768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚239475107680768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚242734407174656 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59416770 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72089686 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚267661306641408 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚267661062307328 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚231575668319408 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 % _timestamp‚1.7884992410460181e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131480111 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59416768 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17290195 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚242733479200768 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131506456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3258371520000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3247093191095 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0çZ&‹X2ã- _wandbruntime‚3299  _runtime‚3299ˆõû&äõM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35609175 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59436930 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59436928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚239556744769536 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚616 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59436930 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3260402964480 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚239556744769536 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚267752701352448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚242817147734016 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35609177 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17295905 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3248210464581 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52905698 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚239556726766592 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788499242177279e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚242816219746816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚234902913417893 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚234901988403325 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 _runtime‚3299.955968324 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 271583744 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131551473 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3259492980224 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131551473 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3247302489373 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17296521 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3408491815424 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52905080 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131525102 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚239556726766592 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚231654685913952 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚264344209537024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3260402964480 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚2643441855969289Šd~åúM EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚616 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59436930 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3260402964480 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚239556744769536 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚267752701352448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚242817147734016 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35609177 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17295905 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3248210464581 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52905698 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚239556726766592 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788499242177279e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚242816219746816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚234902913417893 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚234901988403325 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 _runtime‚3299.955968324 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 271583744 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131551473 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3259492980224 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131551473 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3247302489373 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17296521 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3408491815424 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52905080 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131525102 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚239556726766592 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚231654685913952 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚264344209537024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3260402964480 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚264344185596928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35609175 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59436930 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59436928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚239556744769536 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0èÆ¿ ‡2æ- _wandbruntime‚3299  _runtime‚3299ŠZfO\çÖ 2loss/SRqqt3/assignment_loss‚1.1365023845540625 /loss/FRt4/detection_loss‚2.7878906991746693 .loss/FBt3/detection_loss‚4.651958733797073 0loss/FRt4/assignment_loss‚3.4410805613906295 2loss/SRqqt1/assignment_loss‚1.6436490147773708 0loss/FBt2/assignment_loss‚0.2234748631271941 0loss/SRqqt1/detection_loss‚2.224103296441691 0loss/SRqqt4/detection_loss‚2.492453444462556 % _timestamp‚1.7884992421801968e+09 2loss/SRqqt2/assignment_loss‚1.7025394732398647 /loss/FRt1/detection_loss‚1.6096571749761044 /loss/FRt3/detection_loss‚1.9319002817747957 _step‚233 epoch‚2 0loss/FRt2/assignment_loss‚3.1803076332030087 1loss/FBt3/assignment_loss‚0.26215958657849114 1loss/FBt4/assignment_loss‚0.04270397728153815 /loss/FRt2/detection_loss‚2.5616476471009464 0loss/SRqqt2/detection_loss‚2.436755965862955 2loss/SRqqt4/assignment_loss‚1.9503801943591008 trainer/global_step‚2849 0loss/FRt1/assignment_loss‚1.9744698954563515 0loss/FRt3/assignment_loss‚1.8376329894793235 1loss/SRqqt3/detection_loss‚1.2972743886567297 /loss/FBt1/detection_loss‚0.8674226385738814 %loss/total_loss‚46.91538813695822 _runtime‚ 3299.95735009 1loss/FBt1/assignment_loss‚0.16169683901471954 /loss/FBt2/detection_loss‚3.0137380295329623 .loss/FBt4/detection_loss‚3.483988424142202þü´baèÛ trainer/global_step‚2849 0loss/FRt1/assignment_loss‚1.9744698954563515 0loss/FRt3/assignment_loss‚1.8376329894793235 1loss/SRqqt3/detection_loss‚1.2972743886567297 /loss/FBt1/detection_loss‚0.8674226385738814 %loss/total_loss‚46.91538813695822 _runtime‚ 3299.95735009 1loss/FBt1/assignment_loss‚0.16169683901471954 /loss/FBt2/detection_loss‚3.0137380295329623 .loss/FBt4/detection_loss‚3.483988424142202 2loss/SRqqt3/assignment_loss‚1.1365023845540625 /loss/FRt4/detection_loss‚2.7878906991746693 .loss/FBt3/detection_loss‚4.651958733797073 0loss/FRt4/assignment_loss‚3.4410805613906295 2loss/SRqqt1/assignment_loss‚1.6436490147773708 0loss/FBt2/assignment_loss‚0.2234748631271941 0loss/SRqqt1/detection_loss‚2.224103296441691 0loss/SRqqt4/detection_loss‚2.492453444462556 % _timestamp‚1.7884992421801968e+09 2loss/SRqqt2/assignment_loss‚1.7025394732398647 /loss/FRt1/detection_loss‚1.6096571749761044 /loss/FRt3/detection_loss‚1.9319002817747957 _step‚233 epoch‚2 0loss/FRt2/assignment_loss‚3.1803076332030087 1loss/FBt3/assignment_loss‚0.26215958657849114 1loss/FBt4/assignment_loss‚0.04270397728153815 /loss/FRt2/detection_loss‚2.5616476471009464 0loss/SRqqt2/detection_loss‚2.436755965862955 2loss/SRqqt4/assignment_loss‚1.9503801943591008éå/,“é‚(:ˆ ªªéÔ›Þºm gpu.0.gpu‚16.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚117.193' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.29825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öÊüÇiê‚(:Þ ªªéÔíÊÀo! disk.sdb2.out‚362.55224609375# disk.nvme0n1.out‚20845.01953125 disk.md0.in‚ 1.94140625 disk.sdb1.in‚0$ memory_percent‚95.96829414367676 proc.cpu.threads‚17 disk.dm-0.out‚ 39465.96875 disk.sdb2.in‚ 0.26953125 network.recv‚ 1.108959e+06 disk.nvme0n1.in‚ 58.9921875 disk.nvme1n1.in‚ 9.62109375* proc.memory.percent‚18.395352363586426 disk.sda1.out‚0! disk.sda2.out‚362.55224609375' proc.memory.availableMB‚ 660.5546875 disk.rbd2.out‚ 0.845703125 network.sent‚ 6.196293e+06 disk.sda.out‚362.55224609375 disk.rbd0.out‚ 0.783203125 disk.md0p1.in‚ 1.94140625 disk.md0p1.out‚ 348.5078125 disk.dm-0.in‚ 68.61328125 disk.sda2.in‚1.671875 disk.md0.out‚ 348.5078125 disk.rbd0.in‚0 disk.sdb.in‚ 0.26953125 cpu‚25.15070895992577 disk.sdb1.out‚0 disk.sdb.out‚362.55224609375 disk.rbd2.in‚0$ disk./.usageGB‚2995.937976837158# disk.nvme1n1.out‚18620.94921875 disk.sda.in‚1.671875# proc.memory.rssMB‚ 3013.89453125( disk./.usagePercent‚83.7676797298229 disk.sda1.in‚0°P¾n“ë‚(:ˆ ¹ªéÔŠœàk gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚125.603' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.40075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 jükXì‚(:Í ¹ªéÔ™‘˜m disk.sdb1.out‚0 disk.md0.out‚349.875$ memory_percent‚95.96989154815674( proc.memory.availableMB‚ 660.29296875( disk./.usagePercent‚83.7676797298229 disk.rbd0.out‚ 0.783203125 disk.nvme1n1.in‚ 9.62109375 disk.sda.out‚ 363.970703125 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.md0p1.in‚ 1.94140625 disk.dm-0.out‚ 39675.09375 disk.sda2.in‚1.671875 network.sent‚ 6.290111e+06 disk.sda.in‚1.671875 disk.sda2.out‚ 363.970703125 disk.md0.in‚ 1.94140625 disk.rbd2.in‚0 disk.sdb.in‚ 0.26953125# proc.memory.rssMB‚ 3013.89453125 proc.cpu.threads‚17$ disk./.usageGB‚2995.937976837158 disk.sdb2.in‚ 0.26953125 disk.sdb2.out‚ 363.970703125 disk.nvme0n1.in‚ 58.9921875* proc.memory.percent‚18.395352363586426" disk.nvme1n1.out‚ 18778.3671875 network.recv‚ 1.11384e+06 disk.sda1.in‚0 disk.md0p1.out‚349.875 disk.sdb.out‚ 363.970703125" disk.nvme0n1.out‚ 20896.7265625 disk.dm-0.in‚ 68.61328125 disk.rbd2.out‚ 0.845703125 cpu‚25.150143539422427¢9Ç”‘í‚(:† ȪéÔ®Ôk gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚173.676' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.419 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±°Ûhî‚(:Ý ȪéÔèÒÖm disk.sdb1.in‚0 disk.md0.out‚ 351.51953125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 0.783203125 disk.sdb.in‚ 0.26953125 network.sent‚ 6.298292e+06$ disk./.usageGB‚2995.937976837158 disk.rbd2.in‚0 disk.rbd2.out‚ 0.845703125( disk./.usagePercent‚83.7676797298229! disk.nvme0n1.out‚ 21000.109375 disk.sda.in‚1.671875 disk.sda.out‚365.6787109375 disk.sdb1.out‚0 disk.md0p1.out‚ 351.51953125 disk.sda2.in‚1.671875$ memory_percent‚95.96998691558838( proc.memory.availableMB‚ 660.27734375* proc.memory.percent‚18.395352363586426 network.recv‚ 1.118479e+06" disk.nvme1n1.out‚ 18935.7109375 disk.md0.in‚ 1.94140625 disk.sdb2.in‚ 0.26953125 disk.sda1.out‚0 disk.dm-0.in‚ 68.61328125 disk.dm-0.out‚ 39935.8203125 cpu‚25.149464517043643 disk.nvme0n1.in‚ 58.9921875 disk.md0p1.in‚ 1.94140625 disk.sda2.out‚365.6787109375# proc.memory.rssMB‚ 3013.89453125 disk.nvme1n1.in‚ 9.62109375 disk.sdb2.out‚365.6787109375 disk.sdb.out‚365.6787109375 proc.cpu.threads‚17‹t“ “ï‚(:ˆ תéÔœÀ­q gpu.0.gpu‚45.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚155.961' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.99025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰Þ#Šfð‚(:Û תéÔÓ‚íu disk.sdb.out‚367.14501953125( proc.memory.availableMB‚ 660.38671875 disk.md0p1.in‚ 1.94140625 disk.sdb.in‚ 0.26953125 disk.md0.out‚352.9375 disk.sda.in‚1.671875 disk.rbd2.out‚ 0.845703125! disk.sdb2.out‚367.14501953125 disk.sda.out‚367.14501953125 disk.dm-0.out‚40301.31640625# disk.nvme1n1.out‚19197.85546875 proc.cpu.threads‚17 network.recv‚ 1.122559e+06 disk.sda1.out‚0 disk.rbd0.in‚0 disk.nvme1n1.in‚ 9.62109375# proc.memory.rssMB‚ 3013.89453125$ disk./.usageGB‚2995.937976837158! disk.sda2.out‚367.14501953125 disk.rbd2.in‚0# memory_percent‚95.9693193435669 network.sent‚ 6.304131e+06 disk.rbd0.out‚ 0.783203125 disk.sdb1.out‚0 disk.sda2.in‚1.671875" disk.nvme0n1.out‚ 21103.4609375 disk.md0.in‚ 1.94140625* proc.memory.percent‚18.395352363586426 cpu‚25.148820973585384( disk./.usagePercent‚83.7676797298229 disk.sdb2.in‚ 0.26953125 disk.sda1.in‚0 disk.sdb1.in‚0 disk.md0p1.out‚352.9375 disk.nvme0n1.in‚ 58.9921875 disk.dm-0.in‚ 68.61328125f”A@2ñ- _wandbruntime‚3354  _runtime‚3354àÝÀÇòÁ & lr-AdamW/pg1‚0.0009909487721117376 & lr-AdamW/pg2‚0.0009909487721117376 trainer/global_step‚2899 % _timestamp‚1.7884992967514555e+09 _runtime‚3354.528685361 _step‚234ÕÞÒÌóÆ _runtime‚3354.528685361 _step‚234 & lr-AdamW/pg1‚0.0009909487721117376 & lr-AdamW/pg2‚0.0009909487721117376 trainer/global_step‚2899 % _timestamp‚1.7884992967514555e+09êú êŸ2ô- _wandbruntime‚3354  _runtime‚3354>Üò'õìO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17580067 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235074048 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246870520916992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚243556061971968 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3315368915968 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53781064 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73308245 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚238829535532171 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚235526548046608 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 _runtime‚3354.529760756 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60424770 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚246870520916992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17580702 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3465546131968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884992967524428e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60424768 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚268765245114880 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3465781206016 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3314458945024 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133759358 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73308245 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚246871448890880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36200362 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3314458945024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73334588 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133733013 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60424768 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 trainer/global_step‚2899 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60424770 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53780427 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚243556061971968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚243556079974912 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚246871448890880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73334588 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3302062471203 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3315368915968 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53781064 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73308245 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚238829535532171 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚235526548046608 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 _runtime‚3354.529760756 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60424770 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚246870520916992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17580702 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3465546131968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884992967524428e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60424768 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚268765245114880 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3465781206016 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3314458945024 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133759358 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73308245 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚246871448890880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36200362 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3314458945024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73334588 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133733013 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60424768 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 trainer/global_step‚2899 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60424770 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53780427 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚243556061971968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚243556079974912 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚246871448890880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73334588 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3302062471203 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246870520916992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚243556061971968 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133804375 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60444930 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚246954189450240 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133778004 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3304087719689 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60444928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3466949942272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60444930 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3315580405248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17586500 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚268855470819328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚235605565641152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚246954189450240 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226672128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3316490389504 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚243637681057792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133778004 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250612224 _step‚236 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73333076 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60444928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚268855494759424 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚2899 % _timestamp‚1.7884992976891098e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚243637699060736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73359445 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133804375 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚246953261463040 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚238909670400201 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚243637699060736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3316490389504 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53798304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36212427 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53798927 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3303179744481 _runtime‚3355.466932058 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3315580405248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0náHß'ùûM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3315580405248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17585879 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚238908745385633 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73333076 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36212425 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚272322194089472 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133804375 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60444930 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚246954189450240 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133778004 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3304087719689 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60444928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3466949942272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60444930 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3315580405248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17586500 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚268855470819328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚235605565641152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚246954189450240 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226672128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3316490389504 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚243637681057792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133778004 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250612224 _step‚236 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73333076 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60444928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚268855494759424 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚2899 % _timestamp‚1.7884992976891098e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚243637699060736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73359445 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133804375 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚246953261463040 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚238909670400201 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚243637699060736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3316490389504 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53798304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36212427 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53798927 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3303179744481 _runtime‚3355.466932058ìÜ…_¹2ú- _wandbruntime‚3355  _runtime‚3355\öK\ûÖ 0loss/SRqqt3/assignment_loss‚1.20937482923524 1loss/FBt2/assignment_loss‚0.12582923033995838 .loss/FBt4/detection_loss‚2.140670508146286 0loss/FRt2/assignment_loss‚3.7787436521053315 .loss/FRt3/detection_loss‚1.945792831209573 1loss/SRqqt3/detection_loss‚0.9809708589226263 /loss/FBt2/detection_loss‚3.1360905832714505 1loss/FBt4/assignment_loss‚0.41494901970976833 /loss/FRt2/detection_loss‚2.4314056167006495 /loss/FRt4/detection_loss‚3.0309531325879306 _runtime‚3355.468836126 epoch‚2 2loss/SRqqt1/assignment_loss‚1.7702600323865492 0loss/FBt3/assignment_loss‚0.0513159234772805 .loss/FRt1/detection_loss‚1.960758957379025 % _timestamp‚1.7884992976918101e+09 0loss/FRt1/assignment_loss‚1.9540637684495825 0loss/FRt3/assignment_loss‚1.8859731262380426 2loss/SRqqt2/assignment_loss‚1.8057813727666205 1loss/SRqqt2/detection_loss‚2.2152395556712974 /loss/FBt3/detection_loss‚3.5236451625823975 trainer/global_step‚2899 _step‚237 /loss/FRt4/assignment_loss‚3.665660832239234 1loss/SRqqt1/detection_loss‚1.9604717598807426 1loss/SRqqt4/assignment_loss‚1.624199580199815 1loss/FBt1/assignment_loss‚0.09861513539666202 1loss/SRqqt4/detection_loss‚2.3207767428578556 /loss/FBt1/detection_loss‚0.6927008001399891 &loss/total_loss‚44.724243011893904ã×aüÛ _step‚237 /loss/FRt4/assignment_loss‚3.665660832239234 1loss/SRqqt1/detection_loss‚1.9604717598807426 1loss/SRqqt4/assignment_loss‚1.624199580199815 1loss/FBt1/assignment_loss‚0.09861513539666202 1loss/SRqqt4/detection_loss‚2.3207767428578556 /loss/FBt1/detection_loss‚0.6927008001399891 &loss/total_loss‚44.724243011893904 0loss/SRqqt3/assignment_loss‚1.20937482923524 1loss/FBt2/assignment_loss‚0.12582923033995838 .loss/FBt4/detection_loss‚2.140670508146286 0loss/FRt2/assignment_loss‚3.7787436521053315 .loss/FRt3/detection_loss‚1.945792831209573 1loss/SRqqt3/detection_loss‚0.9809708589226263 /loss/FBt2/detection_loss‚3.1360905832714505 1loss/FBt4/assignment_loss‚0.41494901970976833 /loss/FRt2/detection_loss‚2.4314056167006495 /loss/FRt4/detection_loss‚3.0309531325879306 _runtime‚3355.468836126 epoch‚2 2loss/SRqqt1/assignment_loss‚1.7702600323865492 0loss/FBt3/assignment_loss‚0.0513159234772805 .loss/FRt1/detection_loss‚1.960758957379025 % _timestamp‚1.7884992976918101e+09 0loss/FRt1/assignment_loss‚1.9540637684495825 0loss/FRt3/assignment_loss‚1.8859731262380426 2loss/SRqqt2/assignment_loss‚1.8057813727666205 1loss/SRqqt2/detection_loss‚2.2152395556712974 /loss/FBt3/detection_loss‚3.5236451625823975 trainer/global_step‚2899íöÂìý‚(:… æªéÔíÔål gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚178.02' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.505 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬– €aþ‚(:Ö æªéÔ÷˜n disk.sdb1.in‚0# proc.memory.rssMB‚ 3013.89453125! disk.sdb2.out‚369.19091796875# disk.nvme1n1.out‚19286.30859375 disk.rbd2.in‚0# memory_percent‚95.9747314453125 cpu‚25.148281030689365 network.sent‚ 6.321317e+06$ disk./.usageGB‚2994.883861541748 disk.sda.in‚1.671875# disk.nvme0n1.out‚21171.64453125 disk.nvme1n1.in‚ 9.62109375 network.recv‚ 1.127807e+06 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sda2.in‚1.671875) disk./.usagePercent‚83.73820622498171 disk.md0p1.in‚ 1.94140625! disk.sda2.out‚369.19091796875* proc.memory.percent‚18.395352363586426 disk.sdb2.in‚ 0.26953125 disk.sda.out‚369.19091796875 disk.sda1.in‚0 disk.nvme0n1.in‚ 58.9921875 disk.md0.in‚ 1.94140625 disk.md0.out‚ 354.90625 disk.sda1.out‚0 disk.dm-0.in‚ 68.61328125 disk.sdb.in‚ 0.26953125 disk.rbd2.out‚ 0.845703125! proc.memory.availableMB‚659.5 disk.rbd0.out‚ 0.783203125 disk.md0p1.out‚ 354.90625 disk.dm-0.out‚ 40457.953125 disk.sdb.out‚369.19091796875 proc.cpu.threads‚178 דÿ‚(:ˆ õªéÔº®‘m gpu.0.gpu‚13.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚181.699' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.42475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•ÙRf€‚(:Û õªéÔÙîm disk.sdb1.in‚0 disk.sda.in‚1.671875 disk.sda1.in‚0 network.recv‚ 1.133015e+06 disk.sda2.in‚1.671875 disk.sda2.out‚ 370.681640625$ memory_percent‚95.97346782684326* proc.memory.percent‚18.395352363586426 proc.cpu.threads‚17) disk./.usagePercent‚83.72346392621344 disk.sdb2.out‚ 370.681640625 disk.md0p1.out‚ 356.3515625 disk.nvme1n1.in‚ 9.62109375 disk.md0.out‚ 356.3515625 disk.rbd0.in‚0 disk.dm-0.in‚ 68.61328125$ disk./.usageGB‚2994.356605529785 disk.sdb2.in‚ 0.26953125 disk.sda.out‚ 370.681640625 disk.dm-0.out‚40570.83984375 disk.md0.in‚ 1.94140625 disk.sdb.out‚ 370.681640625" disk.nvme0n1.out‚ 21284.4296875# disk.nvme1n1.out‚19286.41015625 disk.rbd2.in‚0( proc.memory.availableMB‚ 659.70703125 cpu‚25.147704176562435 disk.rbd0.out‚ 0.783203125 disk.sdb1.out‚0 network.sent‚ 6.415217e+06 disk.md0p1.in‚ 1.94140625 disk.sdb.in‚ 0.26953125 disk.nvme0n1.in‚ 58.9921875 disk.sda1.out‚0 disk.rbd2.out‚ 0.845703125# proc.memory.rssMB‚ 3013.89453125½²G§‘‚(:† „«éÔ¶ú¶k gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚169.61' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²y~k‚‚(:à „«éÔŽÿ½l disk.sda.out‚372.15380859375 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 68.61328125" disk.nvme0n1.out‚ 21328.0859375" disk.nvme1n1.out‚ 19287.6171875# memory_percent‚95.9749698638916' proc.memory.availableMB‚ 659.4609375) disk./.usagePercent‚83.72091996586181% disk./.usageGB‚2994.2656211853027 disk.sda.in‚1.671875 disk.md0.in‚ 1.94140625* proc.memory.percent‚18.395352363586426 disk.sda1.in‚0! disk.sda2.out‚372.15380859375 disk.nvme0n1.in‚ 58.9921875 disk.rbd2.in‚0 disk.rbd2.out‚ 0.845703125 proc.cpu.threads‚17 disk.sda1.out‚0 network.recv‚ 1.137801e+06 disk.sdb.in‚ 0.26953125 disk.nvme1n1.in‚ 9.62109375 cpu‚25.14693033077057 network.sent‚ 6.421179e+06! disk.sdb2.out‚372.15380859375# proc.memory.rssMB‚ 3013.89453125 disk.sdb2.in‚ 0.26953125 disk.rbd0.out‚ 0.783203125 disk.sdb1.in‚0 disk.md0p1.in‚ 1.94140625 disk.dm-0.out‚ 40615.703125 disk.sda2.in‚1.671875 disk.md0p1.out‚ 357.76171875 disk.sdb.out‚372.15380859375 disk.md0.out‚ 357.76171875ƒÛŒ’ƒ‚(:‡ “«éÔü¸¶m gpu.0.gpu‚51.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚165.266' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.3165 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@3cp„‚(:å “«éÔÏ¡Ïm disk.sdb1.out‚0# disk.nvme0n1.out‚21400.66796875 network.sent‚ 6.427042e+06 disk.md0p1.out‚ 359.62109375 disk.nvme1n1.in‚ 9.62109375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.dm-0.in‚ 68.61328125 disk.md0.out‚ 359.62109375) disk./.usagePercent‚83.72360631802383 disk.sda1.in‚0 disk.sdb.in‚ 0.26953125" disk.nvme1n1.out‚ 19287.7109375( proc.memory.availableMB‚ 659.78515625* proc.memory.percent‚18.395352363586426 cpu‚25.14634337485372 disk.md0.in‚ 1.94140625 network.recv‚ 1.141604e+06! disk.sdb2.out‚374.08154296875 disk.nvme0n1.in‚ 58.9921875 disk.rbd2.in‚0 disk.sda.out‚374.08154296875 disk.rbd0.in‚0 disk.sda2.in‚1.671875 disk.sdb2.in‚ 0.26953125 disk.md0p1.in‚ 1.94140625 disk.rbd2.out‚ 0.845703125$ memory_percent‚95.97299098968506# proc.memory.rssMB‚ 3013.89453125 disk.sda.in‚1.671875 disk.rbd0.out‚ 0.783203125 disk.sdb1.in‚0 disk.dm-0.out‚40688.37890625 disk.sdb.out‚374.08154296875! disk.sda2.out‚374.08154296875% disk./.usageGB‚2994.3616981506348Á›Üä2…- _wandbruntime‚3406  _runtime‚3406I)²NƆÀ & lr-AdamW/pg2‚0.0009873394141675347 trainer/global_step‚2949 $ _timestamp‚1.788499348572203e+09 _runtime‚3406.349428681 _step‚238 & lr-AdamW/pg1‚0.0009873394141675347:±‹GˇŠtrainer/global_step‚2949 $ _timestamp‚1.788499348572203e+09 _runtime‚3406.349428681 _step‚238 & lr-AdamW/pg1‚0.0009873394141675347 & lr-AdamW/pg2‚0.0009873394141675347îCoš2ˆ- _wandbruntime‚3406  _runtime‚3406PiáщëO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3524077902848 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMoª˜æ 'nitor.on_train_batch_start/inactive_split.all.current‚630 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚628 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3370546370048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3357939726311 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782592 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61432770 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3358847701311 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61432768 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚247637016263168 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚247637016263168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74579490 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74579490 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276800860962816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚251008490607104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61432770 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚247637034266112 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3524306685440 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135985915 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36803610 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722688 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚251007562633216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚247637034266112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17869520 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚273276530337280 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136012260 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚239477427773808 trainer/global_step‚2949 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74553147 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3371456340992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚239477444813168 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚242835367500119 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚251008490607104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17870148 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚273276554277376à`ËIö'ŠðO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36803610 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722688 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚251007562633216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚247637034266112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17869520 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚273276530337280 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136012260 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚239477427773808 trainer/global_step‚2949 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74553147 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3371456340992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚239477444813168 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚242835367500119 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚251008490607104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17870148 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚273276554277376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3524077902848 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚630 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚628 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3370546370048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3357939726311 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782592 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61432770 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3358847701311 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61432768 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚247637016263168 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚247637016263168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74579490 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74579490 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276800860962816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚251008490607104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61432770 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚247637034266112 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3524306685440 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135985915 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712ïçiÂ2‹- _wandbruntime‚3407  _runtime‚3407þýÓ û&ŒõM UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74577978 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136057277 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚242915502367941 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚247718653351936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61452930 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54691697 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74604347 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3372577814528 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61452928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚247718653351936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚247718635348992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3371667830272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36815677 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17875394 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61452930 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61452928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _runtime‚ 3407.34590952 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74577978 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚251091231166464 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136057277 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚239556445368352 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3359056999589 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚276892255108608 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54691069 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚2949 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36815675 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3359964974797 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _step‚240 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 % _timestamp‚1.7884993495676374e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226672128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚247718635348992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚251090303179264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚242916427382509 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3372577814528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3371667830272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚251091231166464 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3525475126784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3525248454656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2³B¤£š úM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136057277 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚239556445368352 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3359056999589 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚276892255108608 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54691069 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚2949 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36815675 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3359964974797 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _step‚240 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 % _timestamp‚1.7884993495676374e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226672128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚247718635348992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚251090303179264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚242916427382509 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3372577814528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3371667830272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚251091231166464 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3525475126784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3525248454656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74577978 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136057277 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚242915502367941 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚247718653351936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61452930 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54691697 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74604347 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3372577814528 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61452928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚247718653351936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚247718635348992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3371667830272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36815677 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17875394 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61452930 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61452928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _runtime‚ 3407.34590952 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74577978 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚251091231166464ð{ŠÂ2Ž- _wandbruntime‚3407  _runtime‚3407å¿“œYÓ 0loss/FRt2/assignment_loss‚2.9976531619375404 /loss/FRt3/assignment_loss‚2.235898816777814 1loss/FBt2/assignment_loss‚0.22629153846439573 0loss/FBt4/assignment_loss‚0.2977886344413396 .loss/FRt1/detection_loss‚2.084532700716997 0loss/SRqqt4/detection_loss‚2.542123652737716 /loss/FBt1/detection_loss‚0.7467472213527395 /loss/FBt2/detection_loss‚2.4562786359053392 .loss/FRt4/detection_loss‚3.037793379405449 1loss/SRqqt3/detection_loss‚1.1486152835259928 epoch‚2 _step‚241 0loss/FBt3/assignment_loss‚0.2022100661794996 .loss/FBt4/detection_loss‚2.289463388292413 trainer/global_step‚2949 /loss/FRt4/assignment_loss‚3.253440119069198 .loss/FRt2/detection_loss‚2.479089332575148 /loss/FBt3/detection_loss‚2.9086427177701677 _runtime‚3407.347532578 2loss/SRqqt4/assignment_loss‚1.5009805534420342 1loss/SRqqt1/detection_loss‚2.0251772979443725 1loss/SRqqt2/detection_loss‚2.3969016589901666 % _timestamp‚1.7884993495705283e+09 1loss/FBt1/assignment_loss‚0.17485964661475897 &loss/total_loss‚44.408373443327555 /loss/FRt1/assignment_loss‚2.554331784739214 2loss/SRqqt1/assignment_loss‚1.9984495446763255 2loss/SRqqt3/assignment_loss‚1.0657673325676853 .loss/FRt3/detection_loss‚1.790566471795882 2loss/SRqqt2/assignment_loss‚1.9947705034053687>»ûì^Ø trainer/global_step‚2949 /loss/FRt4/assignment_loss‚3.253440119069198 .loss/FRt2/detection_loss‚2.479089332575148 /loss/FBt3/detection_loss‚2.9086427177701677 _runtime‚3407.347532578 2loss/SRqqt4/assignment_loss‚1.5009805534420342 1loss/SRqqt1/detection_loss‚2.0251772979443725 1loss/SRqqt2/detection_loss‚2.3969016589901666 % _timestamp‚1.7884993495705283e+09 1loss/FBt1/assignment_loss‚0.17485964661475897 &loss/total_loss‚44.408373443327555 /loss/FRt1/assignment_loss‚2.554331784739214 2loss/SRqqt1/assignment_loss‚1.9984495446763255 2loss/SRqqt3/assignment_loss‚1.0657673325676853 .loss/FRt3/detection_loss‚1.790566471795882 2loss/SRqqt2/assignment_loss‚1.9947705034053687 0loss/FRt2/assignment_loss‚2.9976531619375404 /loss/FRt3/assignment_loss‚2.235898816777814 1loss/FBt2/assignment_loss‚0.22629153846439573 0loss/FBt4/assignment_loss‚0.2977886344413396 .loss/FRt1/detection_loss‚2.084532700716997 0loss/SRqqt4/detection_loss‚2.542123652737716 /loss/FBt1/detection_loss‚0.7467472213527395 /loss/FBt2/detection_loss‚2.4562786359053392 .loss/FRt4/detection_loss‚3.037793379405449 1loss/SRqqt3/detection_loss‚1.1486152835259928 epoch‚2 _step‚241 0loss/FBt3/assignment_loss‚0.2022100661794996 .loss/FBt4/detection_loss‚2.289463388292413ñnf—“‘‚(:ˆ ¢«éÔ¾„²l gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.777' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.94425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùœêk’‚(:à ¢«éÔ‰šÚl disk.md0p1.in‚ 1.94140625 disk.md0p1.out‚ 361.0234375 disk.sda2.in‚1.671875# disk.nvme1n1.out‚19288.07421875 disk.rbd0.in‚0 disk.nvme1n1.in‚ 9.62109375( proc.memory.availableMB‚ 659.46484375# proc.memory.rssMB‚ 3013.89453125 disk.md0.in‚ 1.94140625 disk.sda.in‚1.671875# disk.nvme0n1.out‚21546.89453125 disk.md0.out‚ 361.0234375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.out‚ 40834.96875 disk.nvme0n1.in‚ 58.9921875 network.sent‚ 6.433356e+06 disk.sdb2.in‚ 0.26953125 disk.rbd0.out‚ 0.783203125 disk.sdb1.out‚0 disk.sdb.out‚375.55712890625 proc.cpu.threads‚17( disk./.usagePercent‚83.7256214554572 disk.rbd2.in‚0! disk.sdb2.out‚375.55712890625 disk.sda.out‚375.55712890625 disk.sdb1.in‚0 disk.dm-0.in‚ 68.61328125! disk.sda2.out‚375.55712890625$ memory_percent‚95.97494602203369 network.recv‚ 1.146371e+06 disk.sdb.in‚ 0.26953125* proc.memory.percent‚18.395352363586426 disk.rbd2.out‚ 0.845703125 cpu‚25.145792656845334$ disk./.usageGB‚2994.433769226074-[%““‚(:ˆ ±«éÔãÄòp gpu.0.gpu‚16.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚134.575' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.64375 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€XÀl”‚(:á ±«éÔؤ¢u network.sent‚ 6.539963e+06 disk.nvme1n1.in‚ 9.62109375 disk.rbd0.in‚0 disk.rbd0.out‚ 0.783203125% disk./.usageGB‚2994.5629844665527 disk.sdb2.in‚ 0.26953125 disk.sdb1.out‚0" disk.nvme0n1.out‚ 21628.8046875 disk.sda.out‚376.94970703125 disk.sda1.out‚0 disk.md0p1.out‚ 362.37109375$ memory_percent‚95.97029685974121 cpu‚25.14524840375502 network.recv‚ 1.151228e+06 disk.md0.out‚ 362.37109375! disk.sdb2.out‚376.94970703125 disk.md0p1.in‚ 1.94140625 disk.sda2.in‚1.671875 disk.sdb.out‚376.94970703125 disk.nvme0n1.in‚ 58.9921875 disk.sda.in‚1.671875 disk.dm-0.out‚ 40916.9765625 disk.sdb.in‚ 0.26953125' proc.memory.availableMB‚ 660.2265625* proc.memory.percent‚18.395352363586426 disk.rbd2.in‚0 disk.sda1.in‚0! disk.sda2.out‚376.94970703125! disk.nvme1n1.out‚ 19288.171875 disk.md0.in‚ 1.94140625 disk.dm-0.in‚ 68.61328125# proc.memory.rssMB‚ 3013.89453125 proc.cpu.threads‚17) disk./.usagePercent‚83.72923436766175 disk.rbd2.out‚ 0.845703125 disk.sdb1.in‚0UÞï~‘•‚(:† À«éÔ³Õm gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚179.246' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.8115 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ´÷k–‚(:à À«éÔ¡´ñm disk.sdb1.out‚0 disk.sda2.in‚1.671875# disk.nvme0n1.out‚21841.72265625 disk.md0.out‚ 364.52734375 disk.sda.out‚379.18017578125$ memory_percent‚95.97337245941162( proc.memory.availableMB‚ 659.72265625* proc.memory.percent‚18.395352363586426 disk.sdb1.in‚0 disk.md0.in‚ 1.94140625 disk.rbd2.in‚0 cpu‚25.144732108906776 proc.cpu.threads‚17 network.sent‚ 6.546105e+06 disk.rbd0.out‚ 0.783203125 disk.dm-0.out‚41130.22265625 disk.sdb.out‚379.18017578125! disk.sda2.out‚379.18017578125 disk.nvme1n1.in‚ 9.62109375! disk.sdb2.out‚379.18017578125% disk./.usageGB‚2994.9377403259277 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb.in‚ 0.26953125 network.recv‚ 1.155661e+06 disk.dm-0.in‚ 68.61328125 disk.nvme0n1.in‚ 58.9921875 disk.sdb2.in‚ 0.26953125 disk.rbd0.in‚0 disk.md0p1.in‚ 1.94140625 disk.rbd2.out‚ 0.845703125 disk.sda.in‚1.671875 disk.md0p1.out‚ 364.52734375# proc.memory.rssMB‚ 3013.89453125) disk./.usagePercent‚83.73971269833739 disk.nvme1n1.out‚19288.5 ™%B2—- _wandbruntime‚3463  _runtime‚3463Ó`,ƒÄ˜¾ % lr-AdamW/pg1‚0.000983633950639853 % lr-AdamW/pg2‚0.000983633950639853 trainer/global_step‚2999 $ _timestamp‚1.788499405718613e+09 _runtime‚3463.495952552 _step‚242N|[8əà % lr-AdamW/pg2‚0.000983633950639853 trainer/global_step‚2999 $ _timestamp‚1.788499405718613e+09 _runtime‚3463.495952552 _step‚242 % lr-AdamW/pg1‚0.000983633950639853ò‰mi<2š- _wandbruntime‚3463  _runtime‚3463VÇSçñ'›ëO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3427543766016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62440768 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚255144604349440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚281370487370240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚281370750578688 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚277787839499776 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75824392 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚255145532323328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚255144604349440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62440770 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚246843049496787 _step‚243 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3426633795072 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55567436 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55566794 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3582911078912 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚ 3463.49773018 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3427543766016 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725§¡"fœðO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75824392 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚255145532323328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚255144604349440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62440770 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚246843049496787 _step‚243 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3426633795072 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55567436 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55566794 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3582911078912 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚ 3463.49773018 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3427543766016 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3427543766016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62440768 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚255144604349440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚281370487370240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚281370750578688 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚277787839499776 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152óBŸ#Y2- _wandbruntime‚3464  _runtime‚3464ªJŠü&žöM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55584624 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚281462133730816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62460930 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚281461881021440 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3427755255296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚255227344895488 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚251799589640192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3427755255296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3583839757312 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62460928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228769280 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚255228272882688 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138310179 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3428665239552 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62460928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18165699 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚277878065204224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3584068526592 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚246923184364817 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138310179 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3414934254697 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚246922259350249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37418927 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚251799607643136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75849249 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ÊN 'ŸûM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚251799607643136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75849249 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55584624 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚281462133730816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62460930 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚281461881021440 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3427755255296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚255227344895488 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚251799589640192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3427755255296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3583839757312 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62460928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228769280 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚255228272882688 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138310179 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3428665239552 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62460928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18165699 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚277878065204224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3584068526592 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚246923184364817 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138310179 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3414934254697 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚246922259350249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37418927 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ô ÞU2 - _wandbruntime‚3464  _runtime‚3464K"5Y¡Ó .loss/FBt2/detection_loss‚2.192281575873494 &loss/total_loss‚44.299464701227876 /loss/FRt4/assignment_loss‚3.560718140431813 1loss/FBt3/assignment_loss‚0.13821715146663005 1loss/SRqqt2/detection_loss‚2.5473489250455583 trainer/global_step‚2999 0loss/FRt2/assignment_loss‚3.1909116586049397 2loss/SRqqt2/assignment_loss‚1.6804370103137833 /loss/SRqqt4/detection_loss‚2.63998829466956 0loss/FBt1/assignment_loss‚0.1431365591685562 0loss/FBt2/assignment_loss‚0.4199616412237219 .loss/FRt4/detection_loss‚3.199717110821179 /loss/FBt1/detection_loss‚0.8573501180455257 0loss/FRt1/assignment_loss‚2.7307981740344656 1loss/SRqqt1/assignment_loss‚1.656998623162508 /loss/FRt3/detection_loss‚1.6497760207857937 .loss/FBt4/detection_loss‚2.249969333410263 0loss/FRt3/assignment_loss‚1.6711725378409028 2loss/SRqqt3/assignment_loss‚1.1760886446178211 0loss/FBt4/assignment_loss‚0.3712013988297258 1loss/SRqqt1/detection_loss‚1.9010456531278548 /loss/FBt3/detection_loss‚2.8097336400638926 epoch‚2 .loss/FRt2/detection_loss‚2.356643903421031 _runtime‚3464.678086973 _step‚245 2loss/SRqqt4/assignment_loss‚2.1454305592924356 /loss/FRt1/detection_loss‚2.0447542398744023 1loss/SRqqt3/detection_loss‚0.9657837871020123 % _timestamp‚1.7884994069008973e+09ÿÂr^¢Ø 0loss/FRt3/assignment_loss‚1.6711725378409028 2loss/SRqqt3/assignment_loss‚1.1760886446178211 0loss/FBt4/assignment_loss‚0.3712013988297258 1loss/SRqqt1/detection_loss‚1.9010456531278548 /loss/FBt3/detection_loss‚2.8097336400638926 epoch‚2 .loss/FRt2/detection_loss‚2.356643903421031 _runtime‚3464.678086973 _step‚245 2loss/SRqqt4/assignment_loss‚2.1454305592924356 /loss/FRt1/detection_loss‚2.0447542398744023 1loss/SRqqt3/detection_loss‚0.9657837871020123 % _timestamp‚1.7884994069008973e+09 .loss/FBt2/detection_loss‚2.192281575873494 &loss/total_loss‚44.299464701227876 /loss/FRt4/assignment_loss‚3.560718140431813 1loss/FBt3/assignment_loss‚0.13821715146663005 1loss/SRqqt2/detection_loss‚2.5473489250455583 trainer/global_step‚2999 0loss/FRt2/assignment_loss‚3.1909116586049397 2loss/SRqqt2/assignment_loss‚1.6804370103137833 /loss/SRqqt4/detection_loss‚2.63998829466956 0loss/FBt1/assignment_loss‚0.1431365591685562 0loss/FBt2/assignment_loss‚0.4199616412237219 .loss/FRt4/detection_loss‚3.199717110821179 /loss/FBt1/detection_loss‚0.8573501180455257 0loss/FRt1/assignment_loss‚2.7307981740344656 1loss/SRqqt1/assignment_loss‚1.656998623162508 /loss/FRt3/detection_loss‚1.6497760207857937 .loss/FBt4/detection_loss‚2.249969333410263õÑü81n£‚(:ã Ï«éÔõÒ¥o disk.rbd0.out‚ 0.783203125 disk.sdb.out‚380.76513671875 disk.md0.in‚ 1.94140625 disk.md0.out‚ 366.04296875 disk.sda.in‚1.671875$ memory_percent‚95.97172737121582# proc.memory.rssMB‚ 3013.89453125$ disk./.usageGB‚2995.440776824951 disk.rbd0.in‚0 disk.nvme1n1.in‚ 9.62109375 disk.rbd2.out‚ 0.845703125 disk.sdb2.in‚ 0.26953125) disk./.usagePercent‚83.75377780938722# disk.nvme1n1.out‚19288.58984375 disk.md0p1.out‚ 366.04296875 disk.dm-0.out‚41282.48046875 disk.nvme0n1.in‚ 58.9921875! disk.sdb2.out‚380.76513671875' proc.memory.availableMB‚ 659.9921875 cpu‚25.144088937605964 network.sent‚ 6.551909e+06 disk.sdb1.in‚0 disk.dm-0.in‚ 68.61328125 disk.sda1.in‚0 network.recv‚ 1.159608e+06! disk.nvme0n1.out‚ 21993.890625 disk.sda1.out‚0* proc.memory.percent‚18.395352363586426 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.sda.out‚380.76513671875 disk.sdb1.out‚0 disk.md0p1.in‚ 1.94140625 disk.sdb.in‚ 0.26953125 disk.sda2.in‚1.671875! disk.sda2.out‚380.76513671875gâq“¤‚(:ˆ Ï«éÔøדn gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.157' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.78925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑE𒥂(:‡ Þ«éÔÔ÷Ìm gpu.0.gpu‚53.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.118' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.0295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë‚on¦‚(:ã Þ«éÔÚ‘èm disk.sda.out‚382.26513671875 disk.rbd0.in‚0 network.recv‚ 1.165548e+06 disk.sdb1.in‚0 disk.sdb.out‚382.26513671875 disk.dm-0.out‚41500.61328125 disk.rbd2.out‚ 0.857421875* proc.memory.percent‚18.395352363586426 disk.sdb2.in‚ 0.26953125! disk.sdb2.out‚382.26513671875 disk.sda1.in‚0 disk.md0p1.in‚ 1.94140625$ memory_percent‚95.97179889678955) disk./.usagePercent‚83.75374869106194% disk./.usageGB‚2995.4397354125977 disk.rbd0.out‚ 0.857421875 disk.dm-0.in‚ 68.61328125 disk.sda2.in‚1.671875! disk.sda2.out‚382.26513671875 cpu‚25.143280561783417 disk.sdb.in‚ 0.26953125 disk.nvme0n1.in‚ 58.9921875 disk.nvme1n1.in‚ 9.62109375 disk.nvme1n1.out‚ 19288.9375 disk.md0.out‚ 367.47265625 disk.sda.in‚1.671875 disk.sdb1.out‚0 disk.md0.in‚ 1.94140625 disk.rbd2.in‚0( proc.memory.availableMB‚ 659.98046875# proc.memory.rssMB‚ 3013.89453125 proc.cpu.threads‚17 network.sent‚ 6.659528e+06 disk.sda1.out‚0 disk.md0p1.out‚ 367.47265625# disk.nvme0n1.out‚22211.675781257–£ p§‚(:å í«éÔ•†Îm disk.sda1.out‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 660.10546875 disk.md0.in‚ 1.94140625! disk.sdb2.out‚383.84814453125 disk.sdb1.out‚0* proc.memory.percent‚18.395352363586426( disk./.usagePercent‚83.7676946622974 disk.sdb.out‚383.84423828125 disk.nvme1n1.in‚ 9.62109375# proc.memory.rssMB‚ 3013.89453125 network.recv‚ 1.169066e+06% disk./.usageGB‚2995.9385108947754 network.sent‚ 6.664833e+06# disk.nvme0n1.out‚22438.09765625 disk.md0.out‚ 369.02734375$ memory_percent‚95.97103595733643 disk.rbd0.out‚ 0.857421875 disk.md0p1.out‚ 369.02734375 disk.sdb2.in‚ 0.26953125 disk.sda.out‚383.85205078125 disk.sda1.in‚0 disk.sdb1.in‚0 cpu‚25.142673209088056 disk.md0p1.in‚ 1.94140625 disk.dm-0.in‚ 68.61328125 disk.sda.in‚1.671875 disk.sdb.in‚ 0.26953125! disk.sda2.out‚383.86376953125 disk.rbd2.out‚ 0.919921875 proc.cpu.threads‚17 disk.dm-0.out‚41727.13671875 disk.sda2.in‚1.671875 disk.nvme0n1.in‚ 58.9921875" disk.nvme1n1.out‚ 19289.0390625 disk.rbd2.in‚0>6jc‘¨‚(:† í«éÔ˜ºåm gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚131.45' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.8625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âu£š“©‚(:ˆ ü«éÔï¾òk gpu.0.gpu‚16.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.383' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.09575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fˆca^ª‚(:Ó ü«éÔ«¨¤o disk.nvme0n1.in‚ 58.9921875 disk.rbd2.out‚ 0.919921875 network.recv‚ 1.173645e+06$ disk./.usageGB‚2995.938564300537 disk.sdb.in‚ 0.26953125 cpu‚25.141972161989013! disk.sdb2.out‚385.61865234375 disk.sda.out‚385.61865234375 disk.dm-0.in‚ 68.61328125 memory_percent‚ 95.9716796875 proc.memory.availableMB‚660# proc.memory.rssMB‚ 3013.89453125 disk.rbd0.out‚ 0.95947265625 disk.sdb1.in‚0 disk.md0p1.in‚ 1.94140625 disk.sdb.out‚385.61865234375" disk.nvme1n1.out‚ 19289.2265625 proc.cpu.threads‚17 disk.md0.out‚ 370.7109375 disk.rbd0.in‚0 disk.sdb2.in‚ 0.26953125 disk.sda1.out‚0 disk.sdb1.out‚0 disk.dm-0.out‚ 41956.5390625 disk.sda2.in‚1.671875 disk.rbd2.in‚0* proc.memory.percent‚18.395352363586426) disk./.usagePercent‚83.76769615554484! disk.sda2.out‚385.61865234375 disk.nvme0n1.out‚ 22667.3125 disk.nvme1n1.in‚ 9.62109375 disk.md0.in‚ 1.94140625 network.sent‚ 6.673036e+06 disk.sda.in‚1.671875 disk.sda1.in‚0 disk.md0p1.out‚ 370.7109375´ø?…2«- _wandbruntime‚3521  _runtime‚3521xɿǬÁ % _timestamp‚1.7884994636869848e+09 _runtime‚3521.464306196 _step‚246 & lr-AdamW/pg1‚0.0009798331537417368 & lr-AdamW/pg2‚0.0009798331537417368 trainer/global_step‚30491åH;Ì­Æ & lr-AdamW/pg1‚0.0009798331537417368 & lr-AdamW/pg2‚0.0009798331537417368 trainer/global_step‚3049 % _timestamp‚1.7884994636869848e+09 _runtime‚3521.464306196 _step‚246öÜ?®- _wandbruntime‚ŽÃy33521  _runtime‚3521b$bŠò'¯ìO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3470602211527 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77042951 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18449690 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77042951 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚259282574039552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚285940423528960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140491719 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63448768 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚255798942848512 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚255798924845568 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3483631191040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚282299124722176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140518064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38010112 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 % _timestamp‚1.7884994636881135e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3482721220096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚247379204267568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _step‚247 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3641298806784 trainer/global_step‚3049 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63448770 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63448768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚250849806479095 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚619 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77069294 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚255798942848512 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚247379187228208 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚250848881464735 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3469694236527 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚282299100782080 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3641084704256 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259281646065664 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚259282574039552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚3521.465803542 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3482721220096 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56459800 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640âÝ÷'°ñO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3470602211527 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77042951 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18449690 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77042951 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚259282574039552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚285940423528960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140491719 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63448768 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚255798942848512 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚255798924845568 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3483631191040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚282299124722176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140518064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38010112 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 % _timestamp‚1.7884994636881135e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3482721220096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚247379204267568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _step‚247 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3641298806784 trainer/global_step‚3049 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63448770 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63448768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚250849806479095 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚619 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77069294 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚255798942848512 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚247379187228208 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚250848881464735 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3469694236527 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚282299100782080 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3641084704256 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259281646065664 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚259282574039552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚3521.465803542 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3482721220096 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56459800 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640÷À O2±- _wandbruntime‚3522  _runtime‚3522Þò˜“û&²õM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77094151 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚259365314598912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚282389350426624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚250929941347125 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚255880543931392 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63468928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56478307 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3484752664576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3470811509805 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3483842680320 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77067782 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3484752664576 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3642481704960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚259365314598912 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚286031832131584 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140563081 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252709376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚247458221862112 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56477669 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38022175 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856<§©³úM UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 trainer/global_step‚3049 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3483842680320 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252709376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚247458221862112 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56477669 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38022175 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77094151 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚259365314598912 H@DeviceStatsMonitor.on_train_batch_en‰ Æ6Wd/inactive_split.all.current‚638 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚282389350426624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚250929941347125 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚255880543931392 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63468928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56478307 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3484752664576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3470811509805 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3483842680320 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77067782 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3484752664576 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3642481704960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚259365314598912 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚286031832131584 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140563081 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38613360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3699858357760 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚251330083994768 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚263419615755776 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57354028 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚607 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78314196 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57353419 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3700064071680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚263419615755776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3538808645120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18740059 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚254856563461403 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚263418687781888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38613362 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64456770 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚290510244362240 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 _runtime‚3579.444099411 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.788499521666498e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142744621 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680Ò¼pZ_ÄðO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚286810386004480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 _runtime‚3579.444099411 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.788499521666498e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142744621 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18740666 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3525571491635 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3538808645120 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚286810409944576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚254855638447043 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚259879897139712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚263418687781888 _step‚251 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 KEDeviceStatsMoÛCŸš—nitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38613360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3699858357760 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚251330083994768 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚263419615755776 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57354028 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚607 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78314196 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57353419 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3700064071680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚263419615755776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3538808645120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18740059 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚254856563461403 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚263418687781888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38613362 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64456770 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚290510244362240 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38625427 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚598 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18746390 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚286900611708928 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57371219 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3526688764913 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3539930105344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚290601898046464 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚254935773314865 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3527596740121 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚251409084549952 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142789612 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142815983 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚254936698329433 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3540840089600 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚259961516225536 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18745794 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 trainer/global_step‚3099 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚259961516225536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3539930105344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64476930 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 142789612 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64476928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚286900635649024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚290601664211456 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3701262397440 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚251409101589312 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚263502356315136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57371817 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78339053¥S2x'ÇûM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64476930 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 142789612 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64476928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚286900635649024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚290601664211456 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3701262397440 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚251409101589312 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚263502356315136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57371817 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78339053 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38625427 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚598 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18746390 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚286900611708928 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57371219 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3526688764913 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3539930105344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚290601898046464 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚254935773314865 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3527596740121 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚251409084549952 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142789612 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142815983 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚254936698329433 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3540840089600 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚259961516225536 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18745794 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 trainer/global_step‚3099 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚259961516225536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3539930105344ü¡}2È- _wandbruntime‚3580  _runtime‚3580¸%÷ÛZÉÔ 1loss/FBt1/assignment_loss‚0.20316623762422784 1loss/SRqqt1/detection_loss‚2.3199480521051505 trainer/global_step‚3099 % _timestamp‚1.7884995226653485e+09 _runtime‚3580.442323573 2loss/SRqqt3/assignment_loss‚1.1569778951615546 1loss/FBt2/assignment_loss‚0.16321402332201906 1loss/SRqqt2/detection_loss‚2.4290342043186057 /loss/FBt4/detection_loss‚3.6150542497634888 2loss/SRqqt2/assignment_loss‚1.9217298724528016 1loss/SRqqt4/assignment_loss‚1.423332477418276 /loss/FBt1/detection_loss‚0.7224647135511544 _step‚253 /loss/FRt3/assignment_loss‚1.310076569872243 /loss/FRt1/assignment_loss‚2.899769915775819 /loss/FRt4/assignment_loss‚3.745823399016732 0loss/SRqqt3/detection_loss‚0.918676670303504 /loss/FRt2/assignment_loss‚3.590112290117476 /loss/FRt1/detection_loss‚1.9085488970290911 1loss/SRqqt4/detection_loss‚2.5509388538507314 /loss/FRt2/detection_loss‚2.2782946278651557 .loss/FRt4/detection_loss‚3.317523495147103 &loss/total_loss‚44.393544775464086 epoch‚2 0loss/FBt3/assignment_loss‚0.5253289432136322 1loss/FBt4/assignment_loss‚0.23722612870915327 /loss/FRt3/detection_loss‚1.6861668492534332 /loss/FBt2/detection_loss‚2.4214681850539312 /loss/FBt3/detection_loss‚1.8981389734480116 0loss/SRqqt1/assignment_loss‚1.15052925109079\>l‹_ÊÙ % _timestamp‚1.7884995226653485e+09 _runtime‚3580.442323573 2loss/SRqqt3/assignment_loss‚1.1569778951615546 1loss/FBt2/assignment_loss‚0.16321402332201906 1loss/SRqqt2/detection_loss‚2.4290342043186057 /loss/FBt4/detection_loss‚3.6150542497634888 2loss/SRqqt2/assignment_loss‚1.9217298724528016 1loss/SRqqt4/assignment_loss‚1.423332477418276 /loss/FBt1/detection_loss‚0.7224647135511544 _step‚253 /loss/FRt3/assignment_loss‚1.310076569872243 /loss/FRt1/assignment_loss‚2.899769915775819 /loss/FRt4/assignment_loss‚3.745823399016732 0loss/SRqqt3/detection_loss‚0.918676670303504 /loss/FRt2/assignment_loss‚3.590112290117476 /loss/FRt1/detection_loss‚1.9085488970290911 1loss/SRqqt4/detection_loss‚2.5509388538507314 /loss/FRt2/detection_loss‚2.2782946278651557 .loss/FRt4/detection_loss‚3.317523495147103 &loss/total_loss‚44.393544775464086 epoch‚2 0loss/FBt3/assignment_loss‚0.5253289432136322 1loss/FBt4/assignment_loss‚0.23722612870915327 /loss/FRt3/detection_loss‚1.6861668492534332 /loss/FBt2/detection_loss‚2.4214681850539312 /loss/FBt3/detection_loss‚1.8981389734480116 0loss/SRqqt1/assignment_loss‚1.15052925109079 1loss/FBt1/assignment_loss‚0.20316623762422784 1loss/SRqqt1/detection_loss‚2.3199480521051505 trainer/global_step‚3099ý< ’Ë‚(:‡ ǬéÔ»Ö´m gpu.0.gpu‚29.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚118.142' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.5355 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p$_Ì‚(:Ô ǬéÔ±ýåo disk.sda1.out‚0 disk.rbd0.out‚ 1.06201171875& proc.memory.availableMB‚ 660.109375 network.sent‚ 6.817306e+06 disk.sdb.out‚ 394.318359375 disk.nvme0n1.in‚ 58.9921875 disk.sdb2.out‚ 394.318359375 disk.sda.in‚1.671875 disk.sda.out‚ 394.318359375 network.recv‚ 1.202331e+06 disk.dm-0.in‚ 72.61328125# disk.nvme1n1.out‚19290.13671875 disk.md0.in‚ 1.94140625 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚ 1.94140625 disk.rbd2.in‚0* proc.memory.percent‚18.395352363586426 disk.md0p1.out‚ 379.06640625 disk.sda2.in‚1.671875# proc.memory.rssMB‚ 3013.89453125# disk./.usageGB‚2995.68856048584 disk.sda1.in‚0 disk.sdb1.in‚0$ memory_percent‚95.97101211547852 proc.cpu.threads‚17) disk./.usagePercent‚83.76070594425119 disk.sdb.in‚ 0.26953125 disk.md0.out‚ 379.06640625 disk.rbd2.out‚1.125 disk.dm-0.out‚ 43074.65625 disk.sda2.out‚ 394.318359375# disk.nvme0n1.out‚23784.51953125 disk.nvme1n1.in‚ 13.62109375 disk.sdb2.in‚ 0.26953125 cpu‚25.13873822219778å·Ðð“Í‚(:ˆ Ö¬éÔ±Ïïl gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚179.579' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.89475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛÁ–\΂(:Ñ Ö¬éÔª¡èm% disk./.usageGB‚2994.8858604431152 disk.nvme0n1.in‚ 58.9921875 disk.md0.out‚ 380.4296875 disk.rbd0.out‚1.125 disk.sdb1.out‚0 disk.sda2.in‚1.671875$ memory_percent‚95.97270488739014) disk./.usagePercent‚83.73826211510054 disk.sdb2.out‚ 395.748046875 disk.sda1.in‚0 disk.sdb2.in‚ 0.26953125 disk.sda1.out‚0 disk.sdb.out‚ 395.748046875 network.sent‚ 6.914289e+06 disk.md0.in‚ 1.94140625 disk.rbd2.in‚0 disk.sdb.in‚ 0.26953125# proc.memory.rssMB‚ 3013.89453125* proc.memory.percent‚18.395352363586426" disk.nvme0n1.out‚ 23862.2265625 disk.rbd2.out‚1.125 disk.md0p1.in‚ 1.94140625 disk.md0p1.out‚ 380.4296875 disk.dm-0.in‚ 72.61328125 proc.cpu.threads‚17 disk.sda.in‚1.671875 disk.rbd0.in‚0 disk.dm-0.out‚43152.58203125( proc.memory.availableMB‚ 659.83203125 disk.nvme1n1.in‚ 13.62109375 disk.sda.out‚ 395.748046875 disk.sdb1.in‚0 disk.sda2.out‚ 395.748046875 cpu‚25.138261966265908 network.recv‚ 1.208309e+06# disk.nvme1n1.out‚19290.35546875BUc‘Ï‚(:† å¬éÔÊÝŒm gpu.0.gpu‚51.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚168.992' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„!Ž—dЂ(:Ù å¬éÔ¦çÃm cpu‚25.137696408293543 disk.nvme0n1.in‚ 58.9921875# disk.nvme1n1.out‚19290.46484375 disk.md0.out‚ 382.19140625 disk.sda1.in‚0 disk.md0p1.out‚ 382.19140625 disk.sdb.out‚397.57080078125! disk.sda2.out‚397.57080078125$ memory_percent‚95.97036838531494* proc.memory.percent‚18.395352363586426 proc.cpu.threads‚17) disk./.usagePercent‚83.73825283563424 disk.md0p1.in‚ 1.94140625 disk.dm-0.out‚43292.98046875! disk.sdb2.out‚397.57080078125 network.sent‚ 6.917343e+06 disk.sdb1.out‚0 disk.dm-0.in‚ 72.61328125 disk.rbd2.in‚0 disk.sdb2.in‚ 0.26953125 disk.sda2.in‚1.671875# proc.memory.rssMB‚ 3013.89453125 network.recv‚ 1.211873e+06 disk.sdb1.in‚0 disk.sda.out‚397.57080078125 disk.sdb.in‚ 0.26953125 disk.md0.in‚ 1.94140625 disk.rbd2.out‚1.125 disk.sda.in‚1.671875 disk.rbd0.in‚0 disk.rbd0.out‚1.125$ disk./.usageGB‚2994.885528564453! disk.nvme0n1.out‚ 24002.515625 disk.nvme1n1.in‚ 13.62109375 disk.sda1.out‚0( proc.memory.availableMB‚ 660.21484375iJ~Ñ‚(:† ô¬éÔ¨á¹m gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 í:Õy gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼ê*n^Ò‚(:Ó ô¬éÔ°ðm disk.md0.out‚ 383.578125 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚ 1.94140625 disk.dm-0.out‚ 43423.5234375! disk.sdb2.out‚399.01025390625 disk.rbd0.out‚1.125 disk.md0p1.out‚ 383.578125 disk.nvme1n1.in‚ 13.62109375) disk./.usagePercent‚83.73568786315778" disk.nvme1n1.out‚ 19290.9921875$ memory_percent‚95.97299098968506 disk.sdb1.in‚0 disk.dm-0.in‚ 72.61328125# proc.memory.rssMB‚ 3013.89453125 proc.cpu.threads‚17 network.sent‚ 6.923915e+06 disk.sda.in‚1.671875 disk.sdb.out‚399.01025390625* proc.memory.percent‚18.395352363586426 cpu‚25.137126274465672% disk./.usageGB‚2994.7937927246094 disk.rbd2.out‚1.125 disk.sdb2.in‚ 0.26953125 disk.sda1.in‚0 disk.sdb.in‚ 0.26953125! disk.sda2.out‚399.01025390625 disk.rbd2.in‚0 disk.sda.out‚399.01025390625 disk.sda2.in‚1.671875 disk.nvme0n1.out‚ 24132.53125( proc.memory.availableMB‚ 659.78515625 network.recv‚ 1.216786e+06 disk.md0.in‚ 1.94140625 disk.nvme0n1.in‚ 58.9921875-úä2Ó- _wandbruntime‚3631  _runtime‚3631ˬ¸ÇÔÁ % _timestamp‚1.7884995741429498e+09 _runtime‚3631.920233372 _step‚254 & lr-AdamW/pg1‚0.0009719487478579727 & lr-AdamW/pg2‚0.0009719487478579727 trainer/global_step‚3149üõkÌÕÆ & lr-AdamW/pg1‚0.0009719487478579727 & lr-AdamW/pg2‚0.0009719487478579727 trainer/global_step‚3149 % _timestamp‚1.7884995741429498e+09 _runtime‚3631.920233372 _step‚254þ´Q;2Ö- _wandbruntime‚3631  _runtime‚3631ön5ñ'×ëO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3149 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚263960833427968 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚263960851430912 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 145023868 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267555729498112 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚255280963721968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3758672452096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58247142 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65464770 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79559098 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚295080592207360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3581448746743 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚291321671226880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788499574144044e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚3631.921399437 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚263960851430912 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65464770 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144997523 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚267556657472000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19030530 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79532755 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚255280946682608 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 _step‚255 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65464768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3594896070144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79559098 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚295080343678976 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3758897040384 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65464768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39216612 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267556657472000 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3582356721743 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79532755 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚263960833427968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3595806041088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚291321695166976 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚258862395429351 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚258863320443711 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚267555729498112 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19029895 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39216610 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0•Òv-ö'ØðO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚263960851430912 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 145023868 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267555729498112 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚255280963721968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3758672452096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58247142 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65464770 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79559098 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚295080592207360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3581448746743 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚291321671226880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788499574144044e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚3631.921399437 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚263960851430912 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65464770 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144997523 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚267556657472000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19030530 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79532755 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚255280946682608 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 _step‚255 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65464768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3594896070144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79559098 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚295080343678976 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3758897040384 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65464768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39216612 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267556657472000 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3582356721743 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79532755 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚263960833427968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3595806041088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚291321695166976 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚258862395429351 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚258863320443711 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚267555729498112 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19029895 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39216610 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3149 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚263960833427968 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚258943455311741 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39228675 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚264042452513792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 trainer/global_step‚3149 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 145042514 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39228677 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3582566020021 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79583955 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145068885 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚295171723796992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚3632.864979012 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58264227 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚264042452513792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65484930 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚255359981316512 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65484928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79557586 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚295171982797824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3596017530368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19036180 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚267638470044160 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚264042470516736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3583473995229 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚267639398031360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚291411896931328 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58264857 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3760061926400 $ _timestamp‚1.788499575087138e+09¯l«ªÑÛúM B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3596927514624 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocateHÓÆ{/%d‚79583955 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145068885 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚295171723796992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚3632.864979012 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58264227 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚264042452513792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65484930 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚255359981316512 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65484928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79557586 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚295171982797824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3596017530368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19036180 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚267638470044160 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚264042470516736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3583473995229 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚267639398031360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚291411896931328 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58264857 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3760061926400 $ _timestamp‚1.788499575087138e+09 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚264042470516736 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚258943455311741 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39228675 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚264042452513792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 trainer/global_step‚3149 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 145042514 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39228677 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3582566020021 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79583955 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66472768 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚268041805722112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59141149 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 147276770 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80804000 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80804000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19321287 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3637326001851 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66472770 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244334080 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚299650303181312 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚268041805722112 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚259231826409808 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚262870077426019 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚262869152411659 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚271692771214336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚295832980389376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚295832956449280 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3638233976851 _runtime‚3686.732265148í•¡ÄêðO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚262869152411659 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚271692771214336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚295832980389376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚295832956449280 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3638233976851 _runtime‚3686.732265148 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚271692771214336 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚268041787719168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80777657 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66472768 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚268041805722112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59141149 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 147276770 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80804000 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80804000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19321287 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3637326001851 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66472770 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244334080 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚299650303181312 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚268041805722112 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚259231826409808 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚262870077426019 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615ƒûHôç2ë- _wandbruntime‚3687  _runtime‚3687$ÎzUû&ìõM F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147321787 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚271776439747584 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147295416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66492930 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3653014939648 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚259310844004352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚260 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80828857 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66492928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3652104955392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39831925 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚295923206093824 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚268123406804992 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 $ _timestamp‚1.788499630099978e+09 _runtime‚3687.877766688 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚642 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39831927 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147321787 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59158293 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚268123424807936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19327008 trainer/global_step‚3199 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚268123424807936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3638443275129 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚268123406804992 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80802488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200g'GE'íúM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚271776439747584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19326368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228769280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147321787 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚271776439747584 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147295416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66492930 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3653014939648 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚259310844004352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚260 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80828857 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66492928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3652104955392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39831925 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚295923206093824 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚268123406804992 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 $ _timestamp‚1.788499630099978e+09 _runtime‚3687.877766688 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚642 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39831927 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147321787 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59158293 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚268123424807936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19327008 trainer/global_step‚3199 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚268123424807936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3638443275129 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚268123406804992 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80802488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F’ñ‚(:‡ °­éÔ‡Žýl gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.118' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.0295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%ýðïUò‚(:Ê °­éÔ‘Ý´n% disk./.usageGB‚2994.4915657043457 disk.rbd2.out‚1.125 disk.sda1.in‚0# disk.nvme1n1.out‚19292.12890625 disk.rbd0.in‚0 disk.md0p1.out‚389.625 disk.sda2.out‚405.2705078125 disk.nvme0n1.in‚ 58.9921875 disk.nvme1n1.in‚ 13.62109375 disk.md0.in‚ 1.94140625 disk.md0.out‚389.625 disk.dm-0.out‚43824.41015625$ memory_percent‚95.97384929656982# proc.memory.rssMB‚ 3013.89453125 cpu‚25.13537528735714 disk.sdb2.in‚ 0.26953125 disk.sda1.out‚0 disk.rbd0.out‚1.125 disk.md0p1.in‚ 1.94140625 disk.sdb.in‚ 0.26953125 disk.sdb.out‚405.2705078125 disk.sdb1.in‚0 disk.nvme0n1.out‚ 24532.28125 network.sent‚ 7.066153e+06) disk./.usagePercent‚83.72723746917937 disk.rbd2.in‚0( proc.memory.availableMB‚ 659.64453125* proc.memory.percent‚18.395352363586426 proc.cpu.threads‚17 disk.sda.in‚1.671875 disk.sdb1.out‚0 disk.dm-0.in‚ 72.61328125 disk.sda2.in‚1.671875 network.recv‚ 1.243307e+06 disk.sdb2.out‚405.2705078125 disk.sda.out‚405.2705078125<û“ó‚(:ˆ ¿­éÔýêúl gpu.0.gpu‚38.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽãH[ô‚(:Ð ¿­éÔ¿­n* proc.memory.percent‚18.395352363586426 disk.sdb.in‚ 0.26953125 disk.sdb.out‚406.74365234375 disk.nvme1n1.in‚ 13.62109375 disk.sda.in‚1.671875 disk.sda.out‚406.74365234375 disk.sda1.in‚0 cpu‚25.134967638502104) disk./.usagePercent‚83.73973776356245 disk.md0p1.in‚ 1.94140625 disk.md0p1.out‚ 391.03125 disk.dm-0.in‚ 72.61328125 disk.nvme0n1.out‚ 24813.875# disk.nvme1n1.out‚19292.25390625 disk.md0.in‚ 1.94140625 disk.dm-0.out‚44106.12890625$ disk./.usageGB‚2994.938636779785 disk.sdb1.in‚0 disk.rbd2.in‚0! disk.sdb2.out‚406.74365234375 disk.rbd0.in‚0 disk.sda2.in‚1.671875 disk.md0.out‚ 391.03125$ memory_percent‚95.97384929656982 proc.cpu.threads‚17 network.recv‚ 1.249684e+06 disk.rbd0.out‚1.125 disk.sdb1.out‚0 disk.nvme0n1.in‚ 58.9921875 disk.sdb2.in‚ 0.26953125 disk.sda1.out‚0 network.sent‚ 7.163608e+06# proc.memory.rssMB‚ 3013.89453125! disk.sda2.out‚406.74365234375 disk.rbd2.out‚1.125( proc.memory.availableMB‚ 659.64453125¯y–|“õ‚(:ˆ έéÔ¹±ƒm gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚156.911' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.22775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03B%†_ö‚(:Ô έéÔ¶ îm disk.rbd0.out‚1.125 disk.sdb1.out‚0 proc.cpu.threads‚17 network.sent‚ 7.174241e+06 disk.rbd2.in‚0 disk.sdb2.out‚408.7119140625 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sda2.out‚408.7119140625 network.recv‚ 1.26776e+06 disk.nvme1n1.in‚ 13.62109375 disk.dm-0.in‚ 72.61328125$ memory_percent‚95.97392082214355# proc.memory.rssMB‚ 3013.89453125 cpu‚25.134499030665342 disk.nvme0n1.in‚ 58.9921875 disk.sda.in‚1.671875 disk.sda1.out‚0 disk.md0p1.out‚ 392.92578125 disk.sdb.in‚ 0.26953125 disk.sdb.out‚408.7119140625 disk.sda.out‚408.7119140625 disk.dm-0.out‚ 44343.5703125# disk.nvme0n1.out‚25051.08203125 disk.md0.out‚ 392.92578125 disk.sdb2.in‚ 0.26953125 disk.md0p1.in‚ 1.94140625 disk.sda2.in‚1.671875* proc.memory.percent‚18.395352363586426( disk./.usagePercent‚83.7432298293841% disk./.usageGB‚2995.0635299682617 disk.md0.in‚ 1.94140625' proc.memory.availableMB‚ 659.6328125# disk.nvme1n1.out‚19292.48828125 disk.rbd2.out‚1.125 disk.sda1.in‚0Ø¢)Ä÷‚(:† Ý­éÔœ»ùl gpu.0.gpu‚21.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAÛêÝzllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚121.876' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.469 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QÖaø‚(:Ö Ý­éÔ̵¬m disk.rbd2.in‚0 network.recv‚ 1.271135e+06 disk.sda1.out‚0 disk.md0p1.in‚ 1.94140625 proc.cpu.threads‚17! disk.sdb2.out‚410.30322265625 disk.md0p1.out‚ 394.453125 disk.dm-0.out‚ 44421.203125 disk.sdb.in‚ 0.26953125 disk.nvme0n1.in‚ 58.9921875# disk.nvme0n1.out‚25128.60546875$ memory_percent‚95.97394466400146( proc.memory.availableMB‚ 659.62890625 disk.md0.in‚ 1.94140625 disk.sdb1.in‚0 disk.sdb2.in‚ 0.26953125# proc.memory.rssMB‚ 3013.89453125 cpu‚25.134036056002056 disk.rbd0.in‚0 disk.nvme1n1.in‚ 13.62109375# disk.nvme1n1.out‚19292.59765625) disk./.usagePercent‚83.76076151438842 disk.rbd2.out‚1.125 disk.md0.out‚ 394.453125 disk.rbd0.out‚1.125 disk.sdb.out‚410.30322265625* proc.memory.percent‚18.395352363586426 disk.sda2.in‚1.671875! disk.sda2.out‚410.30322265625 disk.sdb1.out‚0% disk./.usageGB‚2995.6905479431152 disk.sda.in‚1.671875 disk.sda.out‚410.30322265625 disk.sda1.in‚0 disk.dm-0.in‚ 72.61328125 network.sent‚ 7.179394e+06§œ~2ù- _wandbruntime‚3744  _runtime‚3744ÀV±ÇúÁ _step‚262 & lr-AdamW/pg1‚0.0009636927685672985 & lr-AdamW/pg2‚0.0009636927685672985 trainer/global_step‚3249 % _timestamp‚1.7884996866899598e+09 _runtime‚3744.467202772§º†ÊÌûÆ _runtime‚3744.467202772 _step‚262 & lr-AdamW/pg1‚0.0009636927685672985 & lr-AdamW/pg2‚0.0009636927685672985 trainer/global_step‚3249 % _timestamp‚1.7884996866899598e+09†;ÕÞ2ü- _wandbruntime‚3744  _runtime‚37445øì ò'ýìO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚275830740904448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚275829812930560 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3707980891136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82022559 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 trainer/global_step‚3249 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚275829812930560 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67480770 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚304220363163648 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60035214 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚266876834408327 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚300344241671680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19612104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819840 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚266875909393967 _step‚263 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3876121491968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 % _timestamp‚1.7884996866910534e+09 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3694111231959 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3693203256959 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19612725 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚300344265611776 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82048902 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚275829812930560 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3707980891136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82022559 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 trainer/global_step‚3249 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚275829812930560 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67480770 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚304220363163648 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60035214 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚266876834408327 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚300344241671680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19612104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819840 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚266875909393967 _step‚263 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3876121491968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 % _timestamp‚1.7884996866910534e+09 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3694111231959 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3693203256959 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19612725 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚300344265611776 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82048902 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149574689 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82073759 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67500930 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3695228505445 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67500930 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3709102364672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3708192380416 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67500928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149574689 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚3745.498826041 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3708192380416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3877288986112 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚266956969276357 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60053581 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚304312004877312 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574976 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚272204379099136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚266956044261789 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82047390 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 _step‚264 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚272204379099136 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚272204361096192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚272204361096192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19617793 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3877513561088 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60052968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚611 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3694320530237 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚275913481463808 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚613 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40435175 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788499687720844e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚3249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67500928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚275913481463808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚263261723731552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚304311756362240 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149548318 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚272204379099136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚266956044261789 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82047390 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 _step‚264 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚272204379099136 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚272204361096192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚272204361096192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19617793 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3877513561088 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60052968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚611 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3694320530237 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚275913481463808 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚613 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40435175 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788499687720844e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚3249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67500928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚275913481463808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚263261723731552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚304311756362240 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149548318 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149574689 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82073759 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67500930 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3695228505445 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67500930 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3709102364672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3708192380416 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67500928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149574689 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚3745.498826041 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3708192380416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3877288986112 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚266956969276357ˆø¨52‚- _wandbruntime‚3745  _runtime‚3745`ÈZ[ƒÕ /loss/FRt4/assignment_loss‚4.440762926550472 1loss/FBt2/assignment_loss‚0.24195373332032344 epoch‚2 .loss/FBt2/detection_loss‚2.227456635550449 0loss/FRt3/assignment_loss‚1.4887279570102692 1loss/SRqqt4/assignment_loss‚2.025781649209204 /loss/FRt2/detection_loss‚2.7346309274435043 % _timestamp‚1.7884996877236288e+09 /loss/FBt4/detection_loss‚2.1907236330649433 0loss/FRt2/assignment_loss‚3.2959813246360192 1loss/FBt1/assignment_loss‚0.13971914960188256 .loss/FRt1/detection_loss‚2.117534318666347 0loss/SRqqt1/detection_loss‚2.499519308805466 1loss/SRqqt3/detection_loss‚0.8813507574086543 /loss/FBt1/detection_loss‚0.9850875274507237 1loss/SRqqt1/assignment_loss‚1.845250027179718 2loss/SRqqt2/assignment_loss‚1.7370219190639478 2loss/SRqqt3/assignment_loss‚1.2473475295410026 2loss/FBt3/assignment_loss‚0.009026742296290044 .loss/FRt4/detection_loss‚2.697780118269079 0loss/FBt4/assignment_loss‚0.5648199683139811 %loss/total_loss‚44.94271821648114 _runtime‚3745.500633409 0loss/FRt1/assignment_loss‚2.5295174009091146 /loss/FRt3/detection_loss‚1.9055748778896837 1loss/SRqqt2/detection_loss‚2.4452519151899548 trainer/global_step‚3249 1loss/SRqqt4/detection_loss‚2.6405078570048013 .loss/FBt3/detection_loss‚2.051390012105306 _step‚265›*}<`„Ú 1loss/SRqqt3/detection_loss‚0.8813507574086543 /loss/FBt1/detection_loss‚0.9850875274507237 1loss/SRqqt1/assignment_loss‚1.845250027179718 2loss/SRqqt2/assignment_loss‚1.7370219190639478 2loss/SRqqt3/assignment_loss‚1.2473475295410026 2loss/FBt3/assignment_loss‚0.009026742296290044 .loss/FRt4/detection_loss‚2.697780118269079 0loss/FBt4/assignment_loss‚0.5648199683139811 %loss/total_loss‚44.94271821648114 _runtime‚3745.500633409 0loss/FRt1/assignment_loss‚2.5295174009091146 /loss/FRt3/detection_loss‚1.9055748778896837 1loss/SRqqt2/detection_loss‚2.4452519151899548 trainer/global_step‚3249 1loss/SRqqt4/detection_loss‚2.6405078570048013 .loss/FBt3/detection_loss‚2.051390012105306 _step‚265 /loss/FRt4/assignment_loss‚4.440762926550472 1loss/FBt2/assignment_loss‚0.24195373332032344 epoch‚2 .loss/FBt2/detection_loss‚2.227456635550449 0loss/FRt3/assignment_loss‚1.4887279570102692 1loss/SRqqt4/assignment_loss‚2.025781649209204 /loss/FRt2/detection_loss‚2.7346309274435043 % _timestamp‚1.7884996877236288e+09 /loss/FBt4/detection_loss‚2.1907236330649433 0loss/FRt2/assignment_loss‚3.2959813246360192 1loss/FBt1/assignment_loss‚0.13971914960188256 .loss/FRt1/detection_loss‚2.117534318666347 0loss/SRqqt1/detection_loss‚2.499519308805466‰  ÇÉ…‚(:’ ì­éÔ„̶n gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚119.368' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.841999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®‡¿ ]†‚(:Ò ì­éÔóïýn( proc.memory.availableMB‚ 658.08203125 network.recv‚ 1.288381e+06 disk.nvme0n1.in‚ 58.9921875 disk.dm-0.out‚ 44709.1875 disk.sdb.in‚ 0.26953125 disk.sda2.in‚1.671875! disk.sda2.out‚411.89501953125 disk.md0.in‚ 1.94140625 disk.sda1.out‚0 disk.rbd2.out‚1.125 network.sent‚ 7.204758e+06! disk.nvme1n1.out‚ 19293.078125 disk.sda.out‚411.89501953125 disk.rbd0.out‚1.125 disk.dm-0.in‚ 72.61328125 disk.md0p1.out‚ 395.9765625 cpu‚25.133613221988284% disk./.usageGB‚2995.9395484924316 disk.md0.out‚ 395.9765625 disk.sdb1.in‚0) disk./.usagePercent‚83.76772367396214 disk.nvme1n1.in‚ 13.62109375 disk.md0p1.in‚ 1.94140625 disk.sdb.out‚411.89501953125 proc.cpu.threads‚17! disk.sdb2.out‚411.89501953125! disk.nvme0n1.out‚ 25416.109375 disk.rbd2.in‚0 disk.sdb1.out‚0$ memory_percent‚95.98338603973389# proc.memory.rssMB‚ 3013.89453125* proc.memory.percent‚18.395352363586426 disk.sdb2.in‚ 0.26953125 disk.sda.in‚1.671875 disk.sda1.in‚0 disk.rbd0.in‚0•š\½“‡‚(:ˆ û­éÔíáÆm gpu.0.gpu‚20.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚125.271' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.31775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(d"Rˆ‚(:Ç û­éÔ榃n proc.cpu.threads‚17 network.sent‚ 7.307119e+06 disk.sdb2.in‚ 0.26953125 disk.sda1.out‚0 disk.md0p1.in‚ 1.94140625 disk.dm-0.in‚ 72.61328125 disk.md0.in‚ 1.94140625 disk.rbd0.out‚1.125 disk.sdb.in‚ 0.26953125 disk.dm-0.out‚44969.90234375 disk.sda2.out‚ 413.88671875 disk.md0.out‚ 397.90625( proc.memory.availableMB‚ 659.73046875* proc.memory.percent‚18.395352363586426) disk./.usagePercent‚83.76769519560006% disk./.usageGB‚2995.9385299682617 disk.md0p1.out‚ 397.90625 disk.sda2.in‚1.671875" disk.nvme1n1.out‚ 19293.1953125 disk.sda.in‚1.671875 disk.sda1.in‚0# proc.memory.rssMB‚ 3013.89453125 disk.sdb.out‚ 413.88671875# disk.nvme0n1.out‚25676.70703125 disk.rbd2.in‚0 disk.rbd0.in‚0 cpu‚25.13322960928181 disk.sdb1.out‚0 disk.nvme0n1.in‚ 58.9921875 disk.sdb2.out‚ 413.88671875 network.recv‚ 1.305331e+06 disk.sdb1.in‚0 disk.nvme1n1.in‚ 13.62109375 disk.rbd2.out‚1.125 disk.sda.out‚ 413.88671875# memory_percent‚95.9733247756958˜˜¬o“‰‚(:ˆ Š®éÔ„Ï¿m gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.891' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.72275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù%)×\Š‚(:Ñ Š®éÔŒ®®n disk.md0.in‚ 1.94140625 disk.rbd2.in‚0 disk.dm-0.out‚ 45230.828125* proc.memory.percent‚18.395352363586426 disk.sda1.in‚0 disk.sdb1.out‚0$ disk./.usageGB‚2995.937828063965 disk.sda2.in‚1.671875 disk.nvme0n1.in‚ 58.9921875 disk.rbd2.out‚1.125 disk.sdb2.out‚ 415.291015625 disk.sda.out‚ 415.291015625) disk./.usagePercent‚83.76767557006215 disk.nvme1n1.in‚ 13.62109375 disk.dm-0.in‚ 72.61328125 network.recv‚ 1.316569e+06( proc.memory.availableMB‚ 659.72265625 network.sent‚ 7.316813e+06 disk.sdb.out‚ 415.291015625# disk.nvme0n1.out‚25937.51171875 disk.sdb1.in‚0 disk.sda1.out‚0 disk.md0p1.in‚ 1.94140625 disk.sdb.in‚ 0.26953125 disk.md0.out‚ 399.24609375# proc.memory.rssMB‚ 3013.89453125 disk.sda2.out‚ 415.291015625 disk.sdb2.in‚ 0.26953125 disk.rbd0.out‚1.125 disk.md0p1.out‚ 399.24609375$ memory_percent‚95.97337245941162 cpu‚25.132831259206338 proc.cpu.threads‚17# disk.nvme1n1.out‚19293.31640625 disk.sda.in‚1.671875 disk.rbd0.in‚0Ò?¥9“‹‚(:ˆ ™®éÔÙË•m gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.501' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.87525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Qk¡6UŒ‚(:Ê ™®éÔ±±Ïm$ memory_percent‚95.97339630126953# proc.memory.rssMB‚ 3013.89453125 disk.dm-0.in‚ 72.61328125 disk.sdb.in‚ 0.26953125 disk.md0.in‚ 1.94140625 disk.md0.out‚ 400.62109375) disk./.usagePercent‚83.76767557006215 disk.sdb1.in‚0 network.sent‚ 7.319867e+06 disk.sdb.out‚ 416.73046875 disk.nvme0n1.in‚ 58.9921875% proc.memory.availableMB‚ 659.71875 disk.dm-0.out‚45428.89453125 disk.sda2.in‚1.671875 disk.sdb2.out‚ 416.73046875 disk.sda1.in‚0 disk.md0p1.out‚ 400.62109375* proc.memory.percent‚18.395352363586426# disk.nvme1n1.out‚19293.57421875 disk.rbd2.in‚0 disk.sdb2.in‚ 0.26953125 disk.sda1.out‚0 disk.sdb1.out‚0 cpu‚25.13245079717636 disk.sda2.out‚ 416.73046875" disk.nvme0n1.out‚ 26135.3203125 disk.nvme1n1.in‚ 13.62109375 proc.cpu.threads‚17 network.recv‚ 1.320196e+06$ disk./.usageGB‚2995.937828063965 disk.sda.in‚1.671875 disk.sda.out‚ 416.73046875 disk.md0p1.in‚ 1.94140625 disk.rbd2.out‚1.125 disk.rbd0.in‚0 disk.rbd0.out‚1.125ïVP2- _wandbruntime‚3802  _runtime‚3802cöÕÇŽÁ % _timestamp‚1.7884997446433222e+09 _runtime‚3802.420564267 _step‚266 & lr-AdamW/pg1‚0.0009594275775059248 & lr-AdamW/pg2‚0.0009594275775059248 trainer/global_step‚3299rØ^ÌÆ _runtime‚3802.420564267 _step‚266 & lr-AdamW/pg1‚0.0009594275775059248 & lr-AdamW/pg2‚0.0009594275775059248 trainer/global_step‚3299 % _timestamp‚1.7884997446433222e+09Š;¥¶2- _wandbruntime‚3802  _runtime‚3802—m¬Qò'‘ìO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚279967782620672 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚308790440358400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3763158345216 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68488770 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83293804 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚267133602903568 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚279967782620672 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚276203714304512 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3749080512067 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚304855550834176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚270882666376275 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19903024 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚622 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83293804 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83267461 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚3802.421779842 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚276203696301568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚267133585864208 % _timestamp‚1.7884997446444793e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 151782574 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3935125469696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚279966854646784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚267 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3764068316160 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚304855526894080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19902404 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚308790676303872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83267461 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60928764 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚270883591390635 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235945472 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212005376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3934913464320 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3749988487067 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68488768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41026362 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚279966854646784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚276203696301568 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3299 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3764068316160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3763158345216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41026360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚620ûW–´) ’ñO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68488768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41026362 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚279966854646784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚276203696301568 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3299 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3764068316160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3763158345216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41026360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚620 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚279967782620672 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚308790440358400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3763158345216 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68488770 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83293804 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚267133602903568 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚279967782620672 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚276203714304512 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3749080512067 NpØFÎ@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚304855550834176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚270882666376275 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19903024 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚622 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83293804 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83267461 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚3802.421779842 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚276203696301568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚267133585864208 % _timestamp‚1.7884997446444793e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 151782574 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3935125469696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚279966854646784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚267 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3764068316160 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚304855526894080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19902404 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚308790676303872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83267461 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60928764 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚270883591390635 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235945472 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212005376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3934913464320 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3749988487067 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248‹´G\î2“- _wandbruntime‚3803  _runtime‚3803¸ úü&”öM JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83292292 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚276285333390336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚276285315387392 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚280049595192832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19908151 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚308881829694464 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚267212603458752 _runtime‚3803.564817853 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83292292 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚304945776538624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚270962801244097 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151827591 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19908783 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚280050523180032 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3936077095936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68508930 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3751105760553 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68508930 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41038425 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚267212620498112 % _timestamp‚1.7884997457870228e+09 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚270963726258665 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚304945752598528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3750197785345 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚3299 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3765189789696 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚308882082403840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3936305865216 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68508928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3764279805440 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68508928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚276285315387392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252709376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151801220 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60946576 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389J¦O'•ûM WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚276285315387392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252709376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151801220 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60946576 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83292292 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚276285333390336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚276285315387392 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚280049595192832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19908151 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚308881829694464 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚267212603458752 _runtime‚3803.564817853 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83292292 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚304945776538624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚270962801244097 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151827591 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19908783 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚280050523180032 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3936077095936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68508930 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3751105760553 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68508930 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41038425 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚267212620498112 % _timestamp‚1.7884997457870228e+09 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚270963726258665 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚304945752598528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3750197785345 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚3299 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3765189789696 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚308882082403840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3936305865216 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68508928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3764279805440 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68508928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634Œ(Þ÷12–- _wandbruntime‚3803  _runtime‚3803þEsÛX—Ò 1loss/SRqqt4/assignment_loss‚1.824316011598477 1loss/SRqqt2/detection_loss‚2.2866219405470223 .loss/FBt1/detection_loss‚0.844816107613345 1loss/SRqqt1/assignment_loss‚1.702601052224636 0loss/FBt2/assignment_loss‚0.7760958333135042 2loss/FBt3/assignment_loss‚0.019038072413422924 /loss/FBt2/detection_loss‚2.4883983170284942 _step‚269 1loss/SRqqt2/assignment_loss‚1.390403373744981 0loss/FBt1/assignment_loss‚0.1998744502173698 .loss/FRt4/detection_loss‚2.943486653525254 %loss/total_loss‚44.36169525148615 0loss/FRt2/assignment_loss‚3.6744431821923507 1loss/FBt4/assignment_loss‚0.28025452533449213 1loss/SRqqt3/detection_loss‚1.2269983097910881 -loss/FBt4/detection_loss‚2.29274688222829 % _timestamp‚1.7884997457895837e+09 0loss/FRt1/assignment_loss‚3.0431199164494225 0loss/FRt3/assignment_loss‚1.6383164793252945 epoch‚2 _runtime‚3803.566545941 0loss/FRt4/assignment_loss‚3.5588235485142676 2loss/SRqqt3/assignment_loss‚0.9205170077187094 -loss/FRt2/detection_loss‚2.20218354143333 1loss/SRqqt4/detection_loss‚2.7237838208675385 trainer/global_step‚3299 /loss/FRt1/detection_loss‚1.8590334623530236 1loss/SRqqt1/detection_loss‚2.3800340557098387 /loss/FRt3/detection_loss‚1.5512656263075768 .loss/FBt3/detection_loss‚2.534523081034422Ü ]˜× 0loss/FBt2/assignment_loss‚0.7760958333135042 2loss/FBt3/assignment_loss‚0.019038072413422924 /loss/FBt2/detection_loss‚2.4883983170284942 _step‚269 1loss/SRqqt2/assignment_loss‚1.390403373744981 0loss/FBt1/assignment_loss‚0.1998744502173698 .loss/FRt4/detection_loss‚2.943486653525254 %loss/total_loss‚44.36169525148615 0loss/FRt2/assignment_loss‚3.6744431821923507 1loss/FBt4/assignment_loss‚0.28025452533449213 1loss/SRqqt3/detection_loss‚1.2269983097910881 -loss/FBt4/detection_loss‚2.29274688222829 % _timestamp‚1.7884997457895837e+09 0loss/FRt1/assignment_loss‚3.0431199164494225 0loss/FRt3/assignment_loss‚1.6383164793252945 epoch‚2 _runtime‚3803.566545941 0loss/FRt4/assignment_loss‚3.5588235485142676 2loss/SRqqt3/assignment_loss‚0.9205170077187094 -loss/FRt2/detection_loss‚2.20218354143333 1loss/SRqqt4/detection_loss‚2.7237838208675385 trainer/global_step‚3299 /loss/FRt1/detection_loss‚1.8590334623530236 1loss/SRqqt1/detection_loss‚2.3800340557098387 /loss/FRt3/detection_loss‚1.5512656263075768 .loss/FBt3/detection_loss‚2.534523081034422 1loss/SRqqt4/assignment_loss‚1.824316011598477 1loss/SRqqt2/detection_loss‚2.2866219405470223 .loss/FBt1/detection_loss‚0.844816107613345 1loss/SRqqt1/assignment_loss‚1.702601052224636 i—Ø‘™‚(:† ¨®éÔõ‹én gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.992' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}Ší×Vš‚(:Ë ¨®éÔÐÙp disk.sda2.out‚ 418.310546875" disk.nvme0n1.out‚ 26268.4609375 disk.md0.out‚ 402.1328125 disk.sdb2.out‚ 418.310546875 disk.sda1.in‚0 disk.sdb1.in‚0$ memory_percent‚95.97632884979248* proc.memory.percent‚18.395352363586426 disk.rbd0.out‚1.125 disk.nvme1n1.in‚ 13.62109375 disk.nvme1n1.out‚ 19293.71875 disk.md0.in‚ 1.94140625 disk.sdb2.in‚ 0.26953125 disk.sda.in‚1.671875 disk.rbd0.in‚0) disk./.usagePercent‚83.75293967092581$ disk./.usageGB‚2995.410800933838 disk.nvme0n1.in‚ 58.9921875 disk.dm-0.out‚ 45562.1796875 cpu‚25.13189052857552 disk.md0p1.in‚ 1.94140625 disk.sdb1.out‚0 network.sent‚ 7.338741e+06 disk.sda.out‚ 418.310546875( proc.memory.availableMB‚ 659.23828125# proc.memory.rssMB‚ 3013.89453125 network.recv‚ 1.324584e+06 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚1.125 disk.sdb.in‚ 0.26953125 proc.cpu.threads‚17 disk.sda2.in‚1.671875 disk.md0p1.out‚ 402.1328125 disk.dm-0.in‚ 72.61328125 disk.sdb.out‚ 418.310546875ؼ£“›‚(:ˆ ·®éÔ‚¶œm gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.219' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.55475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ}BUœ‚(:Ñ ·®éÔà׶m disk.rbd2.in‚0 disk.dm-0.out‚45772.83203125 disk.sda2.out‚420.3291015625 network.recv‚ 1.328989e+06 disk.nvme0n1.in‚ 58.9921875$ disk./.usageGB‚2995.366081237793 disk.nvme1n1.out‚ 19293.96875 disk.sdb2.in‚ 0.26953125 disk.sdb2.out‚420.3291015625 disk.md0p1.in‚ 1.94140625 disk.sda2.in‚1.671875 disk.rbd0.in‚0$ memory_percent‚95.97504138946533# proc.memory.rssMB‚ 3013.89453125) disk./.usagePercent‚83.75168928950775 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb1.out‚0 disk.sdb.out‚420.3291015625 cpu‚25.13148631056371 proc.cpu.threads‚17 disk.md0.out‚ 404.0859375 disk.sda.in‚1.671875 disk.sda.out‚420.3291015625 disk.sdb1.in‚0# disk.nvme0n1.out‚26478.86328125( proc.memory.availableMB‚ 659.44921875 disk.nvme1n1.in‚ 13.62109375 disk.rbd0.out‚1.125 disk.md0p1.out‚ 404.0859375 disk.dm-0.in‚ 72.61328125* proc.memory.percent‚18.395352363586426 disk.rbd2.out‚1.125 disk.sdb.in‚ 0.26953125 network.sent‚ 7.434295e+06 disk.md0.in‚ 1.9MkÍM4140625׎Mð“‚(:ˆ Æ®éÔ«‚°m gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚167.157' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.78925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G9W–_ž‚(:Ô Æ®éÔ³œäm disk.md0p1.in‚ 1.94140625 disk.nvme0n1.in‚ 58.9921875# disk.nvme0n1.out‚26567.12890625 disk.rbd2.in‚0 disk.rbd2.out‚1.125 disk.sda1.out‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 72.61328125 disk.dm-0.out‚ 45933.109375 cpu‚25.131100540821052 disk.sdb2.out‚421.7470703125 disk.sda1.in‚0 disk.sdb.out‚421.7470703125 proc.cpu.threads‚17 network.sent‚ 7.440561e+06( disk./.usagePercent‚83.7242454279323 disk.sdb2.in‚ 0.26953125 disk.sda.out‚421.7470703125 disk.md0.in‚ 1.94140625 disk.md0.out‚ 405.44140625# proc.memory.rssMB‚ 3013.89453125* proc.memory.percent‚18.395352363586426# disk.nvme1n1.out‚19365.98046875 network.recv‚ 1.333549e+06% disk./.usageGB‚2994.3845558166504 disk.rbd0.out‚1.125 disk.sda2.out‚421.7470703125 disk.nvme1n1.in‚ 13.62109375$ memory_percent‚95.97525596618652' proc.memory.availableMB‚ 659.4140625 disk.sda.in‚1.671875 disk.sdb.in‚ 0.26953125 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.md0p1.out‚ 405.44140625 disk.sda2.in‚1.6718754ZŒ“Ÿ‚(:ˆ Õ®éÔØ“³m gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÌŸ»Y ‚(:Î Õ®éÔÔãËm) disk./.usagePercent‚83.72773546720384$ disk./.usageGB‚2994.509376525879 disk.md0p1.out‚ 406.90234375 disk.md0.out‚ 406.90234375 disk.sdb2.in‚ 0.26953125 disk.sdb2.out‚ 423.267578125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.md0p1.in‚ 1.94140625 disk.dm-0.out‚46013.62890625 disk.nvme1n1.out‚ 19446.03125$ memory_percent‚95.97527980804443 disk.sda2.out‚ 423.267578125 disk.nvme1n1.in‚ 13.62109375 disk.rbd2.out‚1.125 cpu‚25.13066083061028 network.recv‚ 1.33697e+06 network.sent‚ 7.443657e+06 disk.sdb.out‚ 423.267578125 disk.sda2.in‚1.671875* proc.memory.percent‚18.395352363586426 disk.rbd0.out‚1.125 disk.sdb1.in‚0# disk.nvme0n1.out‚26567.59765625 disk.md0.in‚ 1.94140625 disk.sda.in‚1.671875 disk.sda.out‚ 423.267578125 disk.sdb1.out‚0 disk.dm-0.in‚ 72.61328125 disk.sdb.in‚ 0.26953125 disk.nvme0n1.in‚ 58.9921875( proc.memory.availableMB‚ 659.41015625# proc.memory.rssMB‚ 3013.89453125 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0•68Ä2¡- _wandbruntime‚3855  _runtime‚3855J–f^Ç¢Á & lr-AdamW/pg1‚0.0009550720976459968 & lr-AdamW/pg2‚0.0009550720976459968 trainer/global_step‚3349 % _timestamp‚1.7884997979313507e+09 _runtime‚3855.708736543 _step‚270Õâ?UÌ£Æ & lr-AdamW/pg2‚0.0009550720976459968 trainer/global_step‚3349 % _timestamp‚1.7884997979313507e+09 _runtime‚3855.708736543 _step‚270 & lr-AdamW/pg1‚0.0009550720976459968Ž ¯“2¤- _wandbruntime‚3855  _runtime‚3855 )s÷ñ'¥ëO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69496770 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84538706 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚280284668595712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61822994 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _step‚271 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3805865742175 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚280284650592768 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3820155741184 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69496768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚309366836056576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚284103896363008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3993535337472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154035476 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚313360347453952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41629610 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚284103896363008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚280284650592768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚271084465591408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚280284668595712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20193382 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚284104824336896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚631 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69496768 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84538706 _runtime‚3855.709691997 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61822361 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237171200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84512363 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84512363 $ _timestamp‚1.788499797932267e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚284104824336896 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3819245770240 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20192751 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚274890348372943 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154009131 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69496770 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚313360608565248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚274889423358583 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261111296 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 trainer/global_step‚3349 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚271084482630768 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3820155741184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570<ÓÊÒö'¦ðO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚309366836056576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚284103896363008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3993535337472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154035476 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚313360347453952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41629610 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚284103896363008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚280284650592768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚271084465591408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚280284668595712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20193382 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚284104824336896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚631 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69496768 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84538706 _runtime‚3855.709691997 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61822361 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237171200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84512363 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84512363 $ _timestamp‚1.788499797932267e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚284104824336896 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3819245770240 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20192751 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚274890348372943 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154009131 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69496770 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚313360608565248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚274889423358583 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261111296 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 trainer/global_step‚3349 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚271084482630768 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3820155741184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69496770 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84538706 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚280284668595712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61822994 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _step‚271 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3805865742175 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚280284650592768 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3820155741184 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69496768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 Īg2§- _wandbruntime‚3856  _runtime‚3856}{í°!¨öM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84537194 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61840230 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20199178 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252709376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚280366287681536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚271163500225312 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚271163483185952 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69516928 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _runtime‚3856.674195916 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41641675 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154054122 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚3349 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84563563 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20198555 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884997988964071e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚284187564896256 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3806983015661 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69516928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3994701007872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚309457037820928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚274969558226405 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69516930 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚280366269678592 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154080493 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154080493 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚280366287681536 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚274970483240973 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154054122 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3806075040453 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_bê¢^Latch_end/active_bytes.large_pool.peak‚ 13002869760 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3994929777152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41641677 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚284187564896256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228769280 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69516930 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389‡q±¤'©ûM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41641677 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚284187564896256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228769280 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69516930 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84537194 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61840230 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20199178 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252709376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚280366287681536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚271163500225312 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚271163483185952 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69516928 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _runtime‚3856.674195916 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41641675 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154054122 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚3349 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84563563 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20198555 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884997988964071e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚284187564896256 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3806983015661 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69516928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3994701007872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚309457037820928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚274969558226405 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69516930 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚280366269678592 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154080493 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154080493 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚280366287681536 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚274970483240973 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154054122 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3806075040453 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3994929777152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚00æ2ª- _wandbruntime‚3856  _runtime‚3856Lv/\«Ö 0loss/FBt1/assignment_loss‚0.1638120071946127 2loss/FBt3/assignment_loss‚0.043254072439264164 1loss/SRqqt3/detection_loss‚1.1589111800043328 % _timestamp‚1.7884997988982012e+09 2loss/SRqqt4/assignment_loss‚2.1671479609794915 &loss/total_loss‚42.877826553866505 .loss/FRt1/assignment_loss‚2.47692422314388 0loss/FRt2/assignment_loss‚2.8078456826326326 2loss/SRqqt2/assignment_loss‚1.9174399301409721 2loss/SRqqt3/assignment_loss‚1.0586108069785518 .loss/FRt1/detection_loss‚2.046645932444712 /loss/FRt2/detection_loss‚2.4541557411231647 /loss/FRt3/detection_loss‚1.6732100752346657 epoch‚2 trainer/global_step‚3349 1loss/SRqqt1/detection_loss‚1.7566492549010686 0loss/SRqqt1/assignment_loss‚1.43185920648809 1loss/FBt4/assignment_loss‚0.16633091799909985 .loss/FBt4/detection_loss‚2.563128173351288 1loss/FBt2/assignment_loss‚0.27276166241790056 /loss/FRt4/detection_loss‚3.0372688359227675 0loss/SRqqt2/detection_loss‚2.741947367787361 /loss/FBt1/detection_loss‚0.7555467211194846 0loss/SRqqt4/detection_loss‚2.471717279404402 /loss/FBt2/detection_loss‚1.9991564825177193 _runtime‚3856.675221121 0loss/FRt3/assignment_loss‚1.5654888196903116 0loss/FRt4/assignment_loss‚3.7774536280796447 /loss/FBt3/detection_loss‚2.3705605918710884 _step‚2730åœ a¬Û 0loss/SRqqt4/detection_loss‚2.471717279404402 /loss/FBt2/detection_loss‚1.9991564825177193 _runtime‚3856.675221121 0loss/FRt3/assignment_loss‚1.5654888196903116 0loss/FRt4/assignment_loss‚3.7774536280796447 /loss/FBt3/detection_loss‚2.3705605918710884 _step‚273 0loss/FBt1/assignment_loss‚0.1638120071946127 2loss/FBt3/assignment_loss‚0.043254072439264164 1loss/SRqqt3/detection_loss‚1.1589111800043328 % _timestamp‚1.7884997988982012e+09 2loss/SRqqt4/assignment_loss‚2.1671479609794915 &loss/total_loss‚42.877826553866505 .loss/FRt1/assignment_loss‚2.47692422314388 0loss/FRt2/assignment_loss‚2.8078456826326326 2loss/SRqqt2/assignment_loss‚1.9174399301409721 2loss/SRqqt3/assignment_loss‚1.0586108069785518 .loss/FRt1/detection_loss‚2.046645932444712 /loss/FRt2/detection_loss‚2.4541557411231647 /loss/FRt3/detection_loss‚1.6732100752346657 epoch‚2 trainer/global_step‚3349 1loss/SRqqt1/detection_loss‚1.7566492549010686 0loss/SRqqt1/assignment_loss‚1.43185920648809 1loss/FBt4/assignment_loss‚0.16633091799909985 .loss/FBt4/detection_loss‚2.563128173351288 1loss/FBt2/assignment_loss‚0.27276166241790056 /loss/FRt4/detection_loss‚3.0372688359227675 0loss/SRqqt2/detection_loss‚2.741947367787361 /loss/FBt1/detection_loss‚0.7555467211194846‘ ±i‘­‚(:† ä®éÔ÷öÍm gpu.0.gpu‚59.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚121.536' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.384 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0As+mb®‚(:× ä®éÔ“Üæm disk.sdb1.out‚0 network.sent‚ 7.449751e+06) disk./.usagePercent‚83.72869647859827 disk.dm-0.in‚ 72.61328125 disk.md0.in‚ 1.94140625 disk.rbd2.out‚1.125 disk.sdb2.in‚ 0.26953125# disk.nvme0n1.out‚26568.62109375 disk.nvme1n1.in‚ 13.62109375 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.md0p1.in‚ 1.94140625$ disk./.usageGB‚2994.543746948242 disk.md0.out‚ 408.9453125 disk.sda.out‚425.37744140625 disk.dm-0.out‚46092.35546875 disk.nvme0n1.in‚ 58.9921875 disk.sda1.out‚0 disk.rbd0.out‚1.125 disk.sdb.out‚425.37744140625 disk.sda2.in‚1.671875! disk.nvme1n1.out‚ 19523.734375! disk.sdb2.out‚425.37744140625$ memory_percent‚95.97413539886475( proc.memory.availableMB‚ 659.59765625 disk.sda.in‚1.671875 cpu‚25.130283167209413 network.recv‚ 1.341664e+06! disk.sda2.out‚425.37744140625 disk.sda1.in‚0# proc.memory.rssMB‚ 3013.89453125* proc.memory.percent‚18.395352363586426 proc.cpu.threads‚17 disk.sdb.in‚ 0.26953125 disk.md0p1.out‚ 408.9453125 Ì®¿‘¯‚(:† ó®éÔò÷™m gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚121.876' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.469 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á¨^ab°‚(:× ó®éÔž›Ñm network.sent‚ 7.559025e+06) disk./.usagePercent‚83.73965211515515 disk.nvme0n1.in‚ 58.9921875 disk.md0.in‚ 1.94140625! disk.sdb2.out‚426.77294921875" disk.nvme0n1.out‚ 26568.9609375 disk.sdb2.in‚ 0.26953125 disk.rbd0.in‚0 disk.rbd0.out‚1.125# proc.memory.rssMB‚ 3013.89453125$ disk./.usageGB‚2994.935573577881# disk.nvme1n1.out‚19673.39453125 disk.rbd2.in‚0 disk.sda.in‚1.671875 disk.sda1.out‚0 disk.md0p1.in‚ 1.94140625$ memory_percent‚95.97413539886475* proc.memory.percent‚18.395352363586426 disk.md0.out‚ 410.2734375 disk.md0p1.out‚ 410.2734375 disk.dm-0.in‚ 72.61328125 disk.sdb.out‚426.77294921875( proc.memory.availableMB‚ 659.59765625 proc.cpu.threads‚17 disk.dm-0.out‚46242.35546875! disk.sda2.out‚426.77294921875 disk.rbd2.out‚1.125 disk.sda.out‚426.77294921875 disk.sdb.in‚ 0.26953125 network.recv‚ 1.34669e+06 disk.nvme1n1.in‚ 13.62109375 disk.sda1.in‚0 disk.sdb1.out‚0 disk.sda2.in‚1.671875 disk.sdb1.in‚0 cpu‚25.129913490318067û½é“±‚(:ˆ ‚¯éÔº»ñl gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚119.091' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.77275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/!H¤W²‚(:Ì ‚¯éÔÿžâm disk.sdb.in‚ 0.26953125 disk.sda2.out‚ 428.255859375 proc.cpu.threads‚17 disk.nvme1n1.in‚ 13.62109375 disk.sdb2.in‚ 0.26953125 disk.rbd0.out‚1.125 disk.dm-0.in‚ 72.61328125 disk.dm-0.out‚46469.40234375! disk.nvme0n1.out‚ 26569.015625# memory_percent‚95.9756851196289 disk.sdb1.out‚0 disk.sda2.in‚1.671875 cpu‚25.12954272288869 disk.sda1.out‚0# proc.memory.rssMB‚ 3013.89453125* proc.memory.percent‚18.395352363586426 network.recv‚ 1.351404e+06 disk.sdb2.out‚ 428.255859375 disk.sda.out‚ 428.255859375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 58.9921875 disk.md0.out‚ 411.68359375 disk.rbd2.in‚0 disk.md0p1.in‚ 1.94140625 disk.md0p1.out‚ 411.68359375 disk.sdb.out‚ 428.255859375 disk.rbd2.out‚1.125 disk.sda1.in‚0 disk.sdb1.in‚0# disk.nvme1n1.out‚19900.38671875 disk.md0.in‚ 1.94140625 network.sent‚ 7.565333e+06) disk./.usagePercent‚83.73961531727157$ disk./.usageGB‚2994.934257507324 disk.sda.in‚1.671875% proc.memory.availableMB‚ 659.34375WYÙ¢2³- _wandbruntime‚3910  _runtime‚3910ºÒ¡Å´¿ % lr-AdamW/pg1‚0.000950627236662989 % lr-AdamW/pg2‚0.000950627236662989 trainer/global_step‚3399 % _timestamp‚1.7884998527930477e+09 _runtime‚3910.570413256 _step‚274Šú•þ浀 % lr-AdamW/pg1‚0.000950627236662989 % lr-AdamW/pg2‚0.000950627236662989 trainer/global_step‚3399 % _timestamp‚1.7884998527930477e+09 _runtime‚3910.570413256 _step‚274’ËÀr}2¶- _wandbruntime‚3910  _runtime‚3910{äÖ÷ð'·êO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85783608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3876243166208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62716859 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚313878097338880 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70504770 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚278897105355251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85757265 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20483363 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚288240938079232 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚317930278923264 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚278896180340891 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚636 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62716223 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819840 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3861742997283 _runtime‚ 3910.57979635 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚284365622886912 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70504768 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70504768 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚288241866053120 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 $ _timestamp‚1.788499852802276e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156288378 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚317930533743104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚288241866053120 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3876243166208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚284365604883968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚313878121278976 _step‚275 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42232860 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185þ÷s>š ¸ïO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚275035362357968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚284365622886912 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70504768 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70504768 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚288241866053120 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 $ _timestamp‚1.788499852802276e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156288378 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚317930533743104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚288241866053120 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3876243166208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚284365604883968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚313878121278976 _step‚275 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42232860 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85783608 QCDeviceStatsMonitor.on_train_b%³¥‘[atch_start/allocated_bytes.all.current‚ 927973888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3876243166208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62716859 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚313878097338880 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70504770 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚278897105355251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85757265 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20483363 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚288240938079232 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚317930278923264 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚278896180340891 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚636 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62716223 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819840 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3861742997283 _runtime‚ 3910.57979635 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62734090 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3861952295561 % _timestamp‚1.7884998537974663e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85808465 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3399 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚278977240223281 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3877364639744 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 156307024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70524930 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156333395 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚288324606612480 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚284447223969792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156333395 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚318021925824512 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3876454655488 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20489804 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62734731 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3862860270769 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚275114379952512 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚278976315208713 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚275114362913152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70524928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚318021679406592 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85808465 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42244927 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚313968323043328 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156307024 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚313968346983424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3911.575542423 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62734090 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3861952295561 % _timestamp‚1.7884998537974663e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85808465 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3399 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚278977240223281 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3877364639744 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 156307024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70524930 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156333395 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚288324606612480 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚284447223969792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156333395 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚318021925824512 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3876454655488 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20489804 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62734731 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3862860270769 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚275114379952512 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚278976315208713 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚275114362913152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70524928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚318021679406592 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85808465 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42244927 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚313968323043328 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156307024 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚313968346983424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3911.575542423 E_¾Ù 1loss/FBt1/assignment_loss‚0.15946506009896003 -loss/FRt4/detection_loss‚3.17841038107872 0loss/SRqqt1/detection_loss‚2.209510339157922 .loss/FBt3/detection_loss‚2.436456659436226 /loss/FBt2/detection_loss‚2.6196623841921487 2loss/SRqqt1/assignment_loss‚2.1666529088148048 2loss/SRqqt3/assignment_loss‚1.1069262702320073 2loss/SRqqt4/assignment_loss‚1.9908585640437462 1loss/FBt2/assignment_loss‚0.13968198063464143 /loss/FRt3/detection_loss‚1.8574747732230301 /loss/FRt1/assignment_loss‚2.561950582833517 /loss/FRt3/assignment_loss‚2.185700955062077 1loss/FBt4/assignment_loss‚0.08560955618502869 % _timestamp‚1.7884998537996194e+09 /loss/FRt1/detection_loss‚1.9472592980309855 1loss/SRqqt4/detection_loss‚2.2665163269814323 %loss/total_loss‚45.71138127361185 1loss/SRqqt2/detection_loss‚2.7118150194485984 _runtime‚3911.576618358 _step‚277 /loss/FRt4/assignment_loss‚3.725775569677353 1loss/SRqqt3/detection_loss‚0.9026665954036756 /loss/FBt1/detection_loss‚0.5600619013360763 /loss/FRt2/assignment_loss‚3.488371467867563 1loss/FBt3/assignment_loss‚0.12247674648901921 .loss/FRt2/detection_loss‚2.254708939946668 /loss/FBt4/detection_loss‚3.3719140128655867 epoch‚2 trainer/global_step‚3399 2loss/SRqqt2/assignment_loss‚1.6614549805720646•ãÌ×L“¿‚(:ˆ ‘¯éÔ駊m gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚183.867' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.96675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fžGø^À‚(:Ó ‘¯éÔª…âm disk.rbd0.out‚1.125 disk.sdb1.out‚0 disk.md0p1.out‚ 413.15234375 disk.sdb2.out‚429.7919921875 disk.sda2.out‚429.7919921875 cpu‚25.12904958896274 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sdb1.in‚0 disk.dm-0.in‚ 72.61328125 disk.dm-0.out‚46577.66015625( proc.memory.availableMB‚ 748.83203125 network.recv‚ 1.355065e+06 disk.sdb.out‚429.7919921875 disk.nvme1n1.in‚ 13.62109375 disk.rbd2.in‚0 disk.sdb2.in‚ 0.26953125* proc.memory.percent‚17.849254608154297) disk./.usagePercent‚83.73973339048062 disk.sda.in‚1.671875 disk.sda.out‚429.7919921875 disk.rbd0.in‚0! disk.nvme1n1.out‚ 20008.390625 disk.md0.in‚ 1.94140625% disk./.usageGB‚2994.9384803771973 disk.sda1.out‚0 disk.md0p1.in‚ 1.94140625 disk.sda2.in‚1.671875 disk.nvme0n1.in‚ 58.9921875 disk.md0.out‚ 413.15234375 disk.sdb.in‚ 0.26953125# disk.nvme0n1.out‚26569.26953125 network.sent‚ 7.568387e+06 disk.rbd2.out‚1.125$ memory_percent‚95.42949199676514! proc.memory.rssMB‚ 2924.421875çÈÒ+Á‚(:…  ¯éÔ¬úÝl gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.528' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q:׳½Â‚(:Ð  ¯éÔÚÍ›m* proc.memory.percent‚17.849254608154297 network.recv‚ 1.360613e+06 disk.md0p1.out‚ 414.99609375 proc.cpu.threads‚17 disk.nvme1n1.out‚ 20060.96875 disk.sda.out‚431.7119140625 disk.md0.out‚ 414.99609375( disk./.usagePercent‚83.7267361646784 disk.sda1.in‚0 disk.md0p1.in‚ 1.94140625 disk.dm-0.in‚ 72.61328125# disk.nvme0n1.out‚26569.33203125 disk.md0.in‚ 1.94140625 disk.sda.in‚1.671875 disk.rbd2.in‚0 disk.sda1.out‚0$ memory_percent‚95.43118476867676' proc.memory.availableMB‚ 748.5546875 disk.dm-0.out‚46630.30078125 disk.sda2.in‚1.671875 disk.rbd2.out‚1.125% disk./.usageGB‚2994.4736366271973 disk.nvme1n1.in‚ 13.62109375 disk.sdb2.out‚431.7119140625 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb.in‚ 0.26953125 disk.sdb.out‚431.7119140625 disk.sda2.out‚431.7119140625! proc.memory.rssMB‚ 2924.421875 cpu¾¾G"ž‚25.12856745428317 disk.sdb2.in‚ 0.26953125 network.sent‚ 7.675734e+06 disk.rbd0.out‚1.125 disk.sdb1.out‚0 disk.nvme0n1.in‚ 58.99218759nFù“Â(:ˆ ¯¯éÔè·´m gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.207' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.55175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒÜðcÄ‚(:Ø ¯¯éÔ¢Û´n disk.sdb2.in‚ 0.26953125 disk.rbd0.out‚1.125 disk.sdb1.in‚0 disk.md0p1.out‚ 416.38671875 disk.sdb1.out‚0 disk.nvme0n1.in‚ 58.9921875 disk.nvme1n1.in‚ 13.62109375 proc.cpu.threads‚17$ disk./.usageGB‚2994.651954650879 disk.sdb.out‚433.16259765625 disk.md0.in‚ 1.94140625( proc.memory.availableMB‚ 748.49609375 disk.sda1.out‚0 disk.dm-0.in‚ 72.61328125 disk.sdb.in‚ 0.26953125# disk.nvme0n1.out‚26569.51171875$ memory_percent‚95.43154239654541* proc.memory.percent‚17.849254608154297 cpu‚25.128066245833605 network.recv‚ 1.364475e+06 disk.dm-0.out‚ 46727.0859375) disk./.usagePercent‚83.73172201125243! disk.sda2.out‚433.16259765625# disk.nvme1n1.out‚20157.57421875 disk.md0.out‚ 416.38671875 disk.sda.in‚1.671875 disk.sda1.in‚0 disk.rbd0.in‚0 disk.md0p1.in‚ 1.94140625! proc.memory.rssMB‚ 2924.421875 network.sent‚ 7.683705e+06! disk.sdb2.out‚433.16259765625 disk.sda.out‚433.16259765625 disk.sda2.in‚1.671875 disk.rbd2.in‚0 disk.rbd2.out‚1.125yÿë“Å‚(:ˆ ¾¯éԕѪm gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.383' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.09575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Dîß-[Æ‚(:Ð ¾¯éÔµ¹œo disk.sdb1.in‚0 disk.dm-0.in‚ 72.61328125 disk.rbd2.out‚1.125 disk.sdb2.out‚ 434.728515625 disk.sda.out‚ 434.728515625 disk.md0p1.in‚ 1.94140625 disk.sdb.out‚ 434.728515625 disk.nvme1n1.in‚ 13.62109375 cpu‚25.127573282777057 network.recv‚ 1.369208e+06 disk.rbd0.out‚1.125 disk.sdb.in‚ 0.26953125! proc.memory.rssMB‚ 2924.421875) disk./.usagePercent‚83.73976325542962 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sda2.in‚1.671875! disk.nvme0n1.out‚ 26643.640625( proc.memory.availableMB‚ 748.36328125* proc.memory.percent‚17.849254608154297 proc.cpu.threads‚17 network.sent‚ 7.687881e+06 disk.md0p1.out‚ 417.87890625 disk.nvme0n1.in‚ 58.9921875 disk.md0.in‚ 1.94140625$ memory_percent‚95.43235301971436% disk./.usageGB‚2994.9395484924316 disk.sda1.in‚0# disk.nvme1n1.out‚20262.54296875 disk.sda.in‚1.671875 disk.sdb1.out‚0 disk.sdb2.in‚ 0.26953125 disk.dm-0.out‚46906.18359375 disk.sda2.out‚ 434.728515625 disk.md0.out‚ 417.87890625 disk.rbd0.in‚0VÞÄ2Ç- _wandbruntime‚3966  _runtime‚3966KÜðbÇÈÁ trainer/global_step‚3449 % _timestamp‚1.7884999087033496e+09 _runtime‚3966.480777293 _step‚278 & lr-AdamW/pg1‚0.0009460939208592684 & lr-AdamW/pg2‚0.0009460939208592684ŒK›ÌÉÆ & lr-AdamW/pg1‚0.0009460939208592684 & lr-AdamW/pg2‚0.0009460939208592684 trainer/global_step‚3449 % _timestamp‚1.7884999087033496e+09 _runtime‚3966.480777293 _step‚278–Ô*]º2Ê- _wandbruntime‚3966  _runtime‚3966âÝ­ò'ËìO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚288446577178112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚631 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚318389382561280 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3932330591232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 _step‚279 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158514935 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚318389406501376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218296832 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292377979795456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚278986242085168 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158541280 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3931420620288 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87002167 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615HÍ´·÷'ÌñO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3931420620288 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87002167 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚288446577178112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚631 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚318389382561280 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3932330591232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 _step‚279 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158514935 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚318389406501376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218296832 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292377979795456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚278986242085168 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158541280 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144—ÿàW²2Í- _wandbruntime‚3967  _runtime‚3967LR3!ÎöM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71532928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158586297 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20781800 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71532930 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚322592122774016 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71532928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚292461648328704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87026998 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42848175 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚288528178260992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚288528196263936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63629975 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20782427 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚318479608265728 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4112274381824 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚282983997205589 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚288528196263936 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158559926 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚322591882647552 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚292461648328704 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216186368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚288528178260992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3932542080512 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63630604 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚292460720341504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3918737525877 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87053367 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3967.667754821 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42848177 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚627 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87026998 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚318479632205824 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240126464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4112490568192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3933452064768 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3933452064768 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚3449 % _timestamp‚1.7884999098886642e+09‚F'ÏûM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚288528196263936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63629975 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20782427 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚318479608265728 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4112274381824 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚282983997205589 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚288528196263936 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158559926 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚322591882647552 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚292461648328704 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216186368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚288528178260992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3932542080512 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63630604 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚292460720341504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3918737525877 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87053367 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3967.667754821 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42848177 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚627 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87026998 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚318479632205824 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240126464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4112490568192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3933452064768 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3933452064768 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚3449 % _timestamp‚1.7884999098886642e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71532928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158586297 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20781800 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71532930 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚322592122774016 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71532928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚292461648328704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87026998 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42848175 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚288528178260992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856˜ȤT2Ð- _wandbruntime‚3967  _runtime‚3967 /õUÑÏ +loss/FBt2/detection_loss‚2.026752168482 %loss/total_loss‚43.87305389575583 /loss/FRt2/assignment_loss‚3.401838101091839 0loss/FRt3/assignment_loss‚1.2605020956295292 0loss/SRqqt1/detection_loss‚2.262243847052256 1loss/SRqqt3/detection_loss‚0.8825590049151246 /loss/FBt1/detection_loss‚0.8841682797478091 /loss/FBt4/detection_loss‚2.1107895407411785 .loss/FRt1/assignment_loss‚1.98015548951096 2loss/SRqqt2/assignment_loss‚1.5891626900434495 0loss/FBt4/assignment_loss‚0.5468964912884581 /loss/FRt2/detection_loss‚2.5148553142235395 /loss/FRt4/detection_loss‚2.9697340528170266 _runtime‚3967.669124797 _step‚281 1loss/SRqqt3/assignment_loss‚0.983682633619329 1loss/FBt1/assignment_loss‚0.24077644378099095 .loss/FRt1/detection_loss‚2.392018565938391 % _timestamp‚1.7884999098918927e+09 2loss/SRqqt1/assignment_loss‚1.2251603791007288 .loss/FBt3/detection_loss‚4.797212898731232 trainer/global_step‚3449 1loss/FBt3/assignment_loss‚0.01346948261683186 1loss/SRqqt2/detection_loss‚2.6066657161712645 0loss/SRqqt4/detection_loss‚2.651745943292495 0loss/FRt4/assignment_loss‚3.4136932611465456 1loss/FBt2/assignment_loss‚0.29868353904622713 /loss/FRt3/detection_loss‚1.4223883471837857 epoch‚2 1loss/SRqqt4/assignment_loss‚1.397899609584842à$%ZÒÔ 0loss/FRt4/assignment_loss‚3.4136932611465456 1loss/FBt2/assignment_loss‚0.29868353904622713 /loss/FRt3/detection_loss‚1.4223883471837857 epoch‚2 1loss/SRqqt4/assignment_loss‚1.397899609584842 +loss/FBt2/detection_loss‚2.026752168482 %loss/total_loss‚43.87305389575583 /loss/FRt2/assignment_loss‚3.401838101091839 0loss/FRt3/assignment_loss‚1.2605020956295292 0loss/SRqqt1/detection_loss‚2.262243847052256 1loss/SRqqt3/detection_loss‚0.8825590049151246 /loss/FBt1/detection_loss‚0.8841682797478091 /loss/FBt4/detection_loss‚2.1107895407411785 .loss/FRt1/assignment_loss‚1.98015548951096 2loss/SRqqt2/assignment_loss‚1.5891626900434495 0loss/FBt4/assignment_loss‚0.5468964912884581 /loss/FRt2/detection_loss‚2.5148553142235395 /loss/FRt4/detection_loss‚2.9697340528170266 _runtime‚3967.669124797 _step‚281 1loss/SRqqt3/assignment_loss‚0.983682633619329 1loss/FBt1/assignment_loss‚0.24077644378099095 .loss/FRt1/detection_loss‚2.392018565938391 % _timestamp‚1.7884999098918927e+09 2loss/SRqqt1/assignment_loss‚1.2251603791007288 .loss/FBt3/detection_loss‚4.797212898731232 trainer/global_step‚3449 1loss/FBt3/assignment_loss‚0.01346948261683186 1loss/SRqqt2/detection_loss‚2.6066657161712645 0loss/SRqqt4/detection_loss‚2.651745943292495™-÷౑Ó‚(:† ͯéÔ£÷ël gpu.0.gpu‚29.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.676' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.419 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·¢œ5[Ô‚(:Ð ͯéÔÑç–n disk.sda.in‚1.671875 disk.sda1.in‚0 disk.dm-0.out‚47194.19140625 disk.rbd2.out‚1.125( proc.memory.availableMB‚ 748.34765625! proc.memory.rssMB‚ 2924.421875 disk.dm-0.in‚ 72.61328125 disk.md0.out‚ 419.7890625 cpu‚25.127045996433647 network.sent‚ 7.706016e+06 disk.sda2.out‚436.7001953125 disk.nvme1n1.in‚ 13.62109375 disk.sdb2.in‚ 0.26953125 disk.rbd0.in‚0# disk.nvme0n1.out‚26721.51171875 disk.md0.in‚ 1.94140625 disk.rbd0.out‚1.125) disk./.usagePercent‚83.74322758951294 disk.sda1.out‚0 disk.sdb.out‚436.7001953125" memory_percent‚95.432448387146 disk.sdb.in‚ 0.26953125" disk.nvme1n1.out‚ 20472.6796875 network.recv‚ 1.372816e+06$ disk./.usageGB‚2995.063449859619 disk.md0p1.in‚ 1.94140625 disk.md0p1.out‚ 419.7890625 disk.sda2.in‚1.671875 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.sda.out‚436.7001953125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.nvme0n1.in‚ 58.9921875* proc.memory.percent‚17.849254608154297 disk.sdb2.out‚436.7001953125µù1ˆ‘Õ‚(:† ܯéÔ¢À m gpu.0.gpu‚13.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚161.864' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æzú^Ö‚(:Ó ܯéÔ…òþm disk.rbd2.in‚0 disk.sda.out‚438.2529296875 cpu‚25.126573879691385 disk.sdb.in‚ 0.26953125 disk.nvme1n1.in‚ 13.62109375 disk.sda.in‚1.671875* proc.memory.percent‚17.849254608154297 disk.rbd0.in‚0 disk.dm-0.out‚ 47429.0703125 disk.sdb.out‚438.2529296875# disk.nvme1n1.out‚20630.03515625 disk.md0.out‚ 421.27734375 disk.rbd2.out‚1.125# memory_percent‚95.4338788986206! proc.memory.rssMB‚ 2924.421875 disk.sda1.in‚0( proc.memory.availableMB‚ 748.11328125 proc.cpu.threads‚17 disk.md0p1.out‚ 421.27734375 disk.sda2.out‚438.2529296875 disk.nvme0n1.in‚ 58.9921875# disk.nvme0n1.out‚26799.03515625 disk.sdb2.in‚ 0.26953125 disk.sdb2.out‚438.2529296875$ disk./.usageGB‚2995.688449859619 disk.sda1.out‚0 disk.md0p1.in‚ 1.94140625 network.sent‚ 7.800634e+06) disk./.usagePercent‚83.76070285109576 disk.rbd0.out‚1.125 disk.sdb1.in‚0 disk.dm-0.in‚ 72.61328125 disk.sda2.in‚1.671875 disk.md0.in‚ 1.94140625 network.recv‚ 1.378464e+06 disk.sdb1.out‚0áñŸ‘ׂ(:† ë¯éÔ—ƒàl gpu.0.gpu‚15.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚177.688' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.422 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ<_daØ‚(:Ö ë¯éÔŸÒ¥m cpu‚25.126111894480008 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.nvme1n1.in‚ 13.62109375! proc.memory.rssMB‚ 2924.421875) disk./.usagePercent‚83.76772143409096 disk.rbd0.out‚1.125 disk.sda2.in‚1.671875& proc.memory.availableMB‚ 748.359375 disk.md0p1.in‚ 1.94140625 disk.dm-0.out‚47611.86328125 disk.sdb.out‚439.72216796875 network.sent‚ 7.808676e+06$ disk./.usageGB‚2995.939468383789 disk.sda.out‚439.72216796875 disk.sdb1.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 72.61328125! disk.sda2.out‚439.72216796875" disk.nvme1n1.out‚ 20735.1484375 disk.rbd2.out‚1.125 disk.md0p1.out‚ 422.69140625 disk.nvme0n1.in‚ 58.9921875 proc.cpu.threads‚17 disk.md0.out‚ 422.69140625# disk.nvme0n1.out‚26876.71484375 disk.sda1.out‚0 disk.sdb2.in‚ 0.26953125! disk.sdb2.out‚439.72216796875 disk.sda.in‚1.671875 network.recv‚ 1.382409e+06 disk.sda1.in‚0 disk.sdb.in‚ 0.26953125 disk.md0.in‚ 1.94140625$ memory_percent‚95.43237686157227* proc.memory.percent‚17.849254608154297H/µ“Ù‚(:ˆ ú¯éÔÉŒ×k gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚141.427' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.35675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z aù_Ú‚(:Ô ú¯éÔ²€åm disk.dm-0.in‚ 72.61328125 disk.rbd2.out‚1.125 disk.sda.in‚1.671875 disk.sda.out‚441.6572265625 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚1.125! proc.memory.rssMB‚ 2924.421875 disk.sdb1.out‚0 disk.dm-0.out‚ 47899.1484375 cpu‚25.125699928221504 disk.sdb2.out‚441.6572265625 disk.sda1.out‚0 network.recv‚ 1.387649e+06 disk.md0p1.in‚ 1.94140625 disk.sda2.in‚1.671875 disk.sda2.out‚441.6572265625# disk.nvme0n1.out‚26954.23828125* proc.memory.percent‚17.849254608154297 network.sent‚ 7.815049e+06$ disk./.usageGB‚2995.938449859619 disk.md0p1.out‚ 424.55859375 disk.sdb.out‚441.6572265625# disk.nvme1n1.out‚20944.91015625 disk.rbd2.in‚0 disk.sdb2.in‚ 0.26953125 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.nvme0n1.in‚ 58.9921875 disk.md0.out‚ 424.55859375 disk.md0.in‚ 1.94140625) disk./.usagePercent‚83.76769295572888 disk.sdb.in‚ 0.26953125 disk.nvme1n1.in‚ 13.62109375$ memory_percent‚95.43540477752686( proc.memory.availableMB‚ 747.86328125ýÖ;µ2Û- _wandbruntime‚4024  _runtime‚4024yÞm"Äܾ & lr-AdamW/pg2‚0.0009414730949710536 trainer/global_step‚3499 " _timestamp‚1.7884999668324e+09 _runtime‚4024.609792617 _step‚282 & lr-AdamW/pg1‚0.0009414730949710536'ÚØ¥ÉÝà _runtime‚4024.609792617 _step‚282 & lr-AdamW/pg1‚0.0009414730949710536 & lr-AdamW/pg2‚0.0009414730949710536 trainer/global_step‚3499 " _timestamp‚1.7884999668324e+09šaOj2Þ- _wandbruntime‚4024  _runtime‚4024Ö+ò'ßìO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _step‚283 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚296515021511680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚322900667783680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3973497507499 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3988418016256 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88273412 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚322900691723776 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88247069 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 241365504 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚3499 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚292527513466368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚662 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4170219158016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3988418016256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚286909694305507 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72520768 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚296515949485568 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚292527513466368 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88247069 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43439362 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72520770 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3987508045312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚296515021511680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚292527531469312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚327070645576192 _runtime‚4024.611242613 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3987508045312 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21068674 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3972589532499 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚292527531469312 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚282937121812368 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64508698 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 % _timestamp‚1.7884999668336112e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21069336 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160767837 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43439360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144ça„ àñO A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72520770 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3987508045312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚296515021511680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚292527531469312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚327070645576192 _runtime‚4024.611242613 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3987508045312 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 SDDeviceStatsMoni0É‚ë!tor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21068674 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3972589532499 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚292527531469312 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚282937121812368 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64508698 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 % _timestamp‚1.7884999668336112e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21069336 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160767837 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43439360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _step‚283 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚296515021511680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚322900667783680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3973497507499 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3988418016256 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88273412 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚322900691723776 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88247069 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 241365504 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚3499 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚292527513466368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚662 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4170219158016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3988418016256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚286909694305507 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72520768 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚296515949485568 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚292527513466368 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88247069 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43439362›’ŠÌ±2á- _wandbruntime‚4025  _runtime‚4025› Δü&âöM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3973706805777 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220380672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚296597762057728 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64526581 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4171391580160 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160812828 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21075154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21074518 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244320768 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚283016122367552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72540930 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160839199 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚292609150555136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160812828 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160839199 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88298269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚286990754187897 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _step‚284 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88271900 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64525943 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43451427 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚292609150555136 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚286989829173329 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 % _timestamp‚1.7884999680190952e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚296598690044928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72540928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72540930 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚322990893488128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚4025.798480006 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88271900 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43451425 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3974614780985 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712¼Ô'ãûM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚327162309008384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚296597762057728 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64526581 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4171391580160 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160812828 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21075154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21074518 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244320768 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚283016122367552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72540930 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160839199 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚292609150555136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160812828 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160839199 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88298269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚286990754187897 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _step‚284 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88271900 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64525943 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43451427 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚292609150555136 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚286989829173329 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 % _timestamp‚1.7884999680190952e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚296598690044928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72540928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72540930 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚322990893488128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚4025.798480006 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88271900 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43451425 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3974614780985 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3973706805777 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220380672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300652063227904 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4043595470336 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 $ _timestamp‚1.788500024270238e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚296608467757568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚300652991201792 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393984 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚286887984500208 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73528768 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65404515 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4029374762607 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚296608485760512 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89491971 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚327411953006080 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73528768 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65403862 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚290916451287815 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚331640852196352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73528770 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4043595470336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚651 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚290917376302175 _step‚287 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89518314 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚296608485760512 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4044505441280 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89518314 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65404515 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4029374762607 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚296608485760512 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89491971 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚327411953006080 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73528768 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65403862 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚290916451287815 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚331640852196352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73528770 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4043595470336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚651 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚290917376302175 _step‚287 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89518314 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚296608485760512 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4044505441280 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89518314 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300652063227904 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4043595470336 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 $ _timestamp‚1.788500024270238e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚296608467757568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚300652991201792 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393984 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚286887984500208 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389Ÿ8/2õ- _wandbruntime‚4083  _runtime‚4083‡»ä&­ööM D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73548928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65421689 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163092101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚327502202650624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4044716930560 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚286967019134112 % _timestamp‚1.7885000252225983e+09 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4030492036093 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4044716930560 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89516802 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4029584060885 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚655 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4045626914816 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚290996586155637 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89543171 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4045626914816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚653 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4230290475008 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21367014 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4230051219968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 163065730 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚296690086843392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89543171 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 263195136 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44054677 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21367667 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 239255040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚4083.000256165 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_splÁU¾žO it_bytes.large_pool.freed‚327502178710528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73548928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚300735731761152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚296690104846336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73548930 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208zøÆ'÷ûM N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4030492036093 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4044716930560 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89516802 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4029584060885 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚655 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4045626914816 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚290996586155637 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89543171 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4045626914816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚653 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4230290475008 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21367014 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4230051219968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 163065730 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚296690086843392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89543171 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 263195136 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44054677 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21367667 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 239255040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚4083.000256165 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚327502178710528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73548928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚300735731761152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚296690104846336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73548930 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73548928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65421689 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163092101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚327502202650624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4044716930560 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚286967019134112 % _timestamp‚1.7885000252225983e+09 ƒÌUQ2ø- _wandbruntime‚4083  _runtime‚4083žh$$\ùÖ /loss/FRt2/assignment_loss‚3.367386140474459 2loss/SRqqt2/assignment_loss‚1.7516607135534286 /loss/FRt1/detection_loss‚1.5966406761524143 0loss/SRqqt3/detection_loss‚1.057501708379889 2loss/SRqqt4/assignment_loss‚1.9775562639299191 1loss/FBt1/assignment_loss‚0.21998825614144474 0loss/FBt4/assignment_loss‚0.4191488566277258 .loss/FBt2/detection_loss‚2.860626588265101 0loss/FRt3/assignment_loss‚1.9133103838506735 1loss/SRqqt4/detection_loss‚2.1752548970674215 /loss/FBt3/detection_loss‚2.6943731606006622 /loss/FBt4/detection_loss‚2.4366308053334556 epoch‚2 trainer/global_step‚3549 $ _timestamp‚1.788500025224817e+09 2loss/SRqqt3/assignment_loss‚1.1401931246561288 1loss/SRqqt1/detection_loss‚2.1910189842024157 %loss/total_loss‚44.18922917005497 /loss/FRt2/detection_loss‚2.4632009740282848 0loss/SRqqt2/detection_loss‚2.468760414123535 1loss/FBt2/assignment_loss‚0.27434177946351057 /loss/FBt1/detection_loss‚0.5807535786769309 _step‚289 1loss/FBt3/assignment_loss‚0.03901955172329963 _runtime‚4083.001813813 0loss/FRt1/assignment_loss‚2.1334515724863325 0loss/FRt4/assignment_loss‚3.6004183193047843 2loss/SRqqt1/assignment_loss‚1.5503785840926632 .loss/FRt3/detection_loss‚2.010607126748787 /loss/FRt4/detection_loss‚3.2670067101716995WQT aúÛ 0loss/FRt3/assignment_loss‚1.9133103838506735 1loss/SRqqt4/detection_loss‚2.1752548970674215 /loss/FBt3/detection_loss‚2.6943731606006622 /loss/FBt4/detection_loss‚2.4366308053334556 epoch‚2 trainer/global_step‚3549 $ _timestamp‚1.788500025224817e+09 2loss/SRqqt3/assignment_loss‚1.1401931246561288 1loss/SRqqt1/detection_loss‚2.1910189842024157 %loss/total_loss‚44.18922917005497 /loss/FRt2/detection_loss‚2.4632009740282848 0loss/SRqqt2/detection_loss‚2.468760414123535 1loss/FBt2/assignment_loss‚0.27434177946351057 /loss/FBt1/detection_loss‚0.5807535786769309 _step‚289 1loss/FBt3/assignment_loss‚0.03901955172329963 _runtime‚4083.001813813 0loss/FRt1/assignment_loss‚2.1334515724863325 0loss/FRt4/assignment_loss‚3.6004183193047843 2loss/SRqqt1/assignment_loss‚1.5503785840926632 .loss/FRt3/detection_loss‚2.010607126748787 /loss/FRt4/detection_loss‚3.2670067101716995 /loss/FRt2/assignment_loss‚3.367386140474459 2loss/SRqqt2/assignment_loss‚1.7516607135534286 /loss/FRt1/detection_loss‚1.5966406761524143 0loss/SRqqt3/detection_loss‚1.057501708379889 2loss/SRqqt4/assignment_loss‚1.9775562639299191 1loss/FBt1/assignment_loss‚0.21998825614144474 0loss/FBt4/assignment_loss‚0.4191488566277258 .loss/FBt2/detection_loss‚2.860626588265101¡uI§†\û‚(:Ñ Å°éÔœðÜm disk.rbd2.out‚1.125 disk.sda.out‚449.7646484375( proc.memory.availableMB‚ 748.11328125 disk.sdb2.out‚449.7646484375 disk.sda2.in‚1.671875 disk.nvme0n1.in‚ 58.9921875* proc.memory.percent‚17.849254608154297 network.sent‚ 7.942271e+06$ disk./.usageGB‚2994.509777069092# disk.nvme1n1.out‚21605.90234375 disk.sdb.in‚ 0.26953125# memory_percent‚95.4338788986206 cpu‚25.12347316277632 network.recv‚ 1.408254e+06) disk./.usagePercent‚83.72774666655972 disk.rbd2.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 72.61328125 disk.dm-0.out‚ 48812.140625 disk.sda2.out‚449.7646484375 disk.md0.in‚ 1.94140625 disk.sda.in‚1.671875 disk.sda1.in‚0 disk.rbd0.out‚1.125 disk.sdb1.in‚0 disk.sdb.out‚449.7646484375 disk.md0.out‚ 432.32421875 disk.sdb2.in‚ 0.26953125 disk.rbd0.in‚0 disk.md0p1.in‚ 1.94140625 proc.cpu.threads‚17 disk.nvme1n1.in‚ 13.62109375 disk.sdb1.out‚0 disk.md0p1.out‚ 432.32421875# disk.nvme0n1.out‚27206.23828125! proc.memory.rssMB‚ 2924.421875¶ø²à’ü‚(:‡ Å°éÔ©‡ n gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚149.782' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.4455 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Íþ¦_ý‚(:Ô Ô°éÔ‚¼l disk.rbd0.out‚1.125 disk.dm-0.in‚ 72.61328125( proc.memory.availableMB‚ 747.84765625 disk.nvme1n1.in‚ 13.62109375 disk.sda.out‚451.35302734375 disk.sdb1.out‚0 disk.dm-0.out‚ 48889.7890625 network.recv‚ 1.4141e+06 disk.sda2.in‚1.671875 disk.nvme0n1.in‚ 58.9921875# disk.nvme1n1.out‚21683.50390625 disk.md0.in‚ 1.94140625 disk.rbd2.in‚0 disk.rbd2.out‚1.125 disk.sdb1.in‚0 network.sent‚ 8.049692e+06( disk./.usagePercent‚83.7312407589315! disk.sda2.out‚451.35302734375 disk.sda1.in‚0 disk.rbd0.in‚0! proc.memory.rssMB‚ 2924.421875* proc.memory.percent‚17.849254608154297 disk.sdb.in‚ 0.26953125 disk.sda1.out‚0# memory_percent‚95.4355001449585% disk./.usageGB‚2994.6347427368164 disk.md0.out‚ 433.84765625 disk.sda.in‚1.671875 disk.md0p1.in‚ 1.94140625 disk.md0p1.out‚ 433.84765625 disk.sdb2.in‚ 0.26953125 proc.cpu.threads‚17 disk.sdb.out‚451.35302734375# disk.nvme0n1.out‚27206.28515625 cpu‚25.12304121071293! disk.sdb2.out‚451.35302734375©‚\~“þ‚(:ˆ Ô°éÔ­©ãl gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚164.649' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.16225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yì8Zÿ‚(:Ï ã°éÔå¥Ôl disk.sdb.in‚ 0.26953125 disk.sda2.in‚1.671875 disk.nvme0n1.in‚ 58.9921875 disk.sdb2.in‚ 0.26953125 proc.cpu.threads‚17% disk./.usageGB‚2994.7946166992188 disk.sda1.out‚0 disk.md0.out‚ 435.71875 disk.sda.in‚1.671875 cpu‚25.122594162483303 disk.md0p1.in‚ 1.94140625 disk.nvme1n1.in‚ 13.62109375 disk.md0.in‚ 1.94140625 disk.sda1.in‚0 disk.rbd0.in‚0! disk.sdb2.out‚453.29443359375 disk.rbd0.out‚1.125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0p1.out‚ 435.71875 disk.dm-0.out‚ 48995.1328125! disk.sda2.out‚453.29443359375" disk.nvme0n1.out‚ 27206.5546875 network.recv‚ 1.41806e+06! disk.nvme1n1.out‚ 21788.578125 disk.sda.out‚453.29443359375! proc.memory.rssMB‚ 2924.421875* proc.memory.percent‚17.849254608154297 network.sent‚ 8.055477e+06 disk.dm-0.in‚ 72.61328125 disk.sdb.out‚453.29443359375 disk.rbd2.in‚0) disk./.usagePercent‚83.73571090183272 disk.rbd2.out‚1.125$ memory_percent‚95.43406963348389( proc.memory.availableMB‚ 748.08203125Zñn ’€‚(:‡ ã°éÔ¸¯ƒm gpu.0.gpu‚58.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚122.486' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.6215 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:Të2- _wandbruntime‚4134  _runtime‚4134.T¨bÄ‚¾ % lr-AdamW/pg1‚0.000931972782870187 % lr-AdamW/pg2‚0.000931972782870187 trainer/global_step‚3599 % _timestamp‚1.7885000762943754e+09 _runtime‚ 4134.07160959 _step‚290œ*À2Ƀà % lr-AdamW/pg2‚0.000931972782870187 trainer/global_step‚3599 % _timestamp‚1.7885000762943754e+09 _runtime‚ 4134.07160959 _step‚290 % lr-AdamW/pg1‚0.000931972782870187¢¦ÿ42„- _wandbruntime‚4134  _runtime‚4134Úø8Íò'…ìO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66298840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚649 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4287800071168 _step‚291 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165273641 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74536770 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚300689422048768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44645860 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722688 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165299986 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4084344042715 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚304789104944128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4288028853760 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚647 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚336211291022336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚336211038299648 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚294924133284483 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304789104944128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4099682895360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4100592866304 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44645862 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90763216 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74536770 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712û¿/Ò›†ñO P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚649 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4287800071168 _step‚291 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165273641 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74536770 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚300689422048768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44645860 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722688 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165299986 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4084344042715 F@DeviceStatsMonitor.Ä‚0\"on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚304789104944128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4288028853760 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚647 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚336211291022336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚336211038299648 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚294924133284483 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304789104944128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4099682895360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4100592866304 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44645862 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90763216 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74536770 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66298840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0£ñl2‡- _wandbruntime‚4135  _runtime‚4135$M½û&ˆõM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74556930 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165318632 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚300771059137536 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4101714339840 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚304872773477376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90788073 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚295004268152513 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165318632 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4100804355584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74556928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4288981460992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 trainer/global_step‚3599 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚336302445393920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66315998 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚332013463932928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74556928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚304871845490176 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4085461315993 $ _timestamp‚1.788500077266531e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90761704 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44657927 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚295003343137945 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4086369291201 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417920 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66316637 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _runtime‚4135.044350394 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165345003 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4100804355584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90788073 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477824 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74556930 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚336302691811840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4101714339840 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚304872773477376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90788073 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚295004268152513 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165318632 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4100804355584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74556928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4288981460992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 trainer/global_step‚3599 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚336302445393920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66315998 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚332013463932928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74556928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚304871845490176 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4085461315993 $ _timestamp‚1.788500077266531e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90761704 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44657927 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚295003343137945 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4086369291201 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417920 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66316637 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _runtime‚4135.044350394 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165345003 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4100804355584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0¤€J2Š- _wandbruntime‚4135  _runtime‚4135 Rä4Z‹Ô /loss/FBt1/detection_loss‚0.7657872299619374 %loss/total_loss‚43.35697744712675 /loss/FRt2/detection_loss‚2.3924519608660444 /loss/FRt2/assignment_loss‚3.688308820491884 /loss/FRt4/assignment_loss‚3.637501447431503 2loss/SRqqt3/assignment_loss‚0.8329441350007665 1loss/FBt2/assignment_loss‚0.26056955239094437 1loss/FBt4/assignment_loss‚0.27032650520148027 0loss/FRt1/assignment_loss‚2.5466124284267426 .loss/FBt3/detection_loss‚2.795241528087192 0loss/FBt3/assignment_loss‚0.4406232019465885 /loss/FRt4/detection_loss‚2.9323518583851476 1loss/SRqqt4/detection_loss‚2.6833331768329325 _step‚293 /loss/FBt2/detection_loss‚2.2057112921839175 epoch‚2 % _timestamp‚1.7885000772684088e+09 2loss/SRqqt2/assignment_loss‚1.7500276129979353 2loss/SRqqt4/assignment_loss‚1.4397684060610259 1loss/SRqqt3/detection_loss‚1.1000955744637448 /loss/FRt3/assignment_loss‚1.466629140231074 /loss/FBt1/assignment_loss‚0.153325910470011 .loss/FRt1/detection_loss‚1.818473458542685 1loss/SRqqt1/detection_loss‚1.9728254439199673 /loss/FBt4/detection_loss‚2.8530987546994138 _runtime‚4135.045371298 trainer/global_step‚3599 1loss/SRqqt1/assignment_loss‚1.399766642600298 /loss/FRt3/detection_loss‚2.0144801802781163 1loss/SRqqt2/detection_loss‚1.9367231856553981Qœs-_ŒÙ 1loss/SRqqt1/assignment_loss‚1.399766642600298 /loss/FRt3/detection_loss‚2.0144801802781163 1loss/SRqqt2/detection_loss‚1.9367231856553981 /loss/FBt1/detection_loss‚0.7657872299619374 %loss/total_loss‚43.35697744712675 /loss/FRt2/detection_loss‚2.3924519608660444 /loss/FRt2/assignment_loss‚3.688308820491884 /loss/FRt4/assignment_loss‚3.637501447431503 2loss/SRqqt3/assignment_loss‚0.8329441350007665 1loss/FBt2/assignment_loss‚0.26056955239094437 1loss/FBt4/assignment_loss‚0.27032650520148027 0loss/FRt1/assignment_loss‚2.5466124284267426 .loss/FBt3/detection_loss‚2.795241528087192 0loss/FBt3/assignment_loss‚0.4406232019465885 /loss/FRt4/detection_loss‚2.9323518583851476 1loss/SRqqt4/detection_loss‚2.6833331768329325 _step‚293 /loss/FBt2/detection_loss‚2.2057112921839175 epoch‚2 % _timestamp‚1.7885000772684088e+09 2loss/SRqqt2/assignment_loss‚1.7500276129979353 2loss/SRqqt4/assignment_loss‚1.4397684060610259 1loss/SRqqt3/detection_loss‚1.1000955744637448 /loss/FRt3/assignment_loss‚1.466629140231074 /loss/FBt1/assignment_loss‚0.153325910470011 .loss/FRt1/detection_loss‚1.818473458542685 1loss/SRqqt1/detection_loss‚1.9728254439199673 /loss/FBt4/detection_loss‚2.8530987546994138 _runtime‚4135.045371298 trainer/global_step‚3599¥½/¾~‘‚(:† ò°éÔŠü¼l gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚121.536' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.384 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿î¼Ž‚(:Ó ò°éÔߧál disk.sda1.out‚0 disk.dm-0.out‚49144.82421875 disk.nvme0n1.in‚ 58.9921875 disk.rbd0.in‚0 disk.md0p1.in‚ 1.94140625 disk.md0p1.out‚ 437.078125 disk.rbd2.in‚0 disk.dm-0.in‚ 72.61328125 disk.nvme1n1.out‚ 21866.21875 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.nvme1n1.in‚ 13.62109375( proc.memory.availableMB‚ 747.79296875 disk.md0.in‚ 1.94140625 disk.md0.out‚ 437.078125 disk.sda.in‚1.671875! disk.sda2.out‚454.71923828125# disk.nvme0n1.out‚27278.60546875! proc.memory.rssMB‚ 2924.421875 cpu‚25.122271365400678 network.sent‚ 8.061745e+06 network.recv‚ 1.422763e+06 é']¢ disk.sda.out‚454.71923828125 disk.rbd0.out‚1.125 disk.sdb.in‚ 0.26953125 disk.sda2.in‚1.671875* proc.memory.percent‚17.849254608154297 disk.sdb2.in‚ 0.26953125! disk.sdb2.out‚454.71923828125 disk.sda1.in‚0 disk.sdb.out‚454.71923828125$ memory_percent‚95.43583393096924) disk./.usagePercent‚83.73964678212855% disk./.usageGB‚2994.9353828430176 disk.rbd2.out‚1.125‰¿‰’‚(:‡ ±éÔè®Ók gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚168.822' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.2055 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J–K4_‚(:Ô ±éÔ‘Çl$ disk./.usageGB‚2994.935085296631 disk.rbd2.in‚0 disk.rbd2.out‚1.125 disk.md0p1.in‚ 1.94140625 disk.nvme1n1.in‚ 13.62109375# disk.nvme1n1.out‚22021.32421875 disk.sda.out‚456.69482421875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.out‚ 49372.078125! disk.sda2.out‚456.69482421875 disk.sda.in‚1.671875 disk.dm-0.in‚ 72.61328125! proc.memory.rssMB‚ 2924.421875! disk.sdb2.out‚456.69482421875 disk.nvme0n1.in‚ 58.9921875# disk.nvme0n1.out‚27350.75390625* proc.memory.percent‚17.849254608154297 disk.md0.in‚ 1.94140625 disk.sdb2.in‚ 0.26953125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚1.125 disk.sdb.out‚456.69482421875 cpu‚25.121895041242396 network.sent‚ 8.168574e+06 network.recv‚ 1.427319e+06) disk./.usagePercent‚83.73963846260703 disk.md0p1.out‚ 438.9921875 disk.sdb.in‚ 0.26953125$ memory_percent‚95.43452262878418 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.in‚1.671875 disk.md0.out‚ 438.9921875' proc.memory.availableMB‚ 748.0078125W¤g’‘‚(:‡ ±éÔàˆÉk gpu.0.gpu‚34.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥ï?Y’‚(:Î ±éÔ·€ãl disk.sda1.in‚0! disk.sda2.out‚458.25439453125 disk.nvme0n1.in‚ 58.9921875& proc.memory.availableMB‚ 640.421875 network.recv‚ 1.43191e+06 disk.dm-0.in‚ 72.61328125 disk.nvme1n1.in‚ 13.62109375 disk.sda2.in‚1.671875" disk.nvme1n1.out‚ 22065.4609375# proc.memory.rssMB‚ 3031.88671875! disk.nvme0n1.out‚ 27355.203125$ disk./.usageGB‚2994.412082672119 disk.rbd0.in‚0 disk.rbd0.out‚1.125 disk.rbd2.out‚1.125 network.sent‚ 8.174661e+06 disk.sdb2.in‚ 0.26953125 disk.sda.out‚458.25439453125 disk.dm-0.out‚ 49420.6640625 disk.sdb.out‚458.25439453125 cpu‚25.12699844536993! disk.sdb2.out‚458.25439453125 disk.sda.in‚1.671875 disk.sda1.out‚0 disk.md0p1.out‚ 440.484375 disk.md0.in‚ 1.94140625 disk.rbd2.in‚0# memory_percent‚96.0911750793457* proc.memory.percent‚18.505167961120605 disk.sdb1.in‚0 disk.md0p1.in‚ 1.94140625 disk.sdb.in‚ 0.26953125 proc.cpu.threads‚17 disk.md0.out‚ 440.484375 disk.sdb1.out‚0) disk./.usagePercent‚83.72501509033209èÔlœ“‚(:‘ Ÿ±éÔåâËm gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZE,ÐW”‚(:Ì Ÿ±éÔʼïm) disk./.usagePercent‚83.72850779611694 disk.sda1.in‚0! disk.sdb2.out‚459.76123046875 disk.rbd0.in‚0 disk.md0p1.out‚ 441.9296875 disk.dm-0.out‚ 49517.4375 disk.rbd2.in‚0 disk.sdb2.in‚ 0.26953125 network.recv‚ 1.436024e+06 disk.sdb1.in‚0 disk.dm-0.in‚ 72.61328125 disk.nvme0n1.in‚ 58.9921875 disk.md0.out‚ 441.9296875 disk.sdb.in‚ 0.26953125! disk.sda2.out‚459.76123046875 disk.nvme0n1.out‚ 27408.03125 disk.nvme1n1.out‚ 22109.40625( proc.memory.availableMB‚ 640.42578125% disk./.usageGB‚2994.5369987487793 disk.sda.out‚459.76123046875 disk.sdb.out‚459.76123046875 disk.nvme1n1.in‚ 13.62109375# proc.memory.rssMB‚ 3031.88671875 proc.cpu.threads‚17 disk.rbd2.out‚1.125 disk.sdb1.out‚0 disk.md0p1.in‚ 1.94140625 disk.sda2.in‚1.671875 disk.sda.in‚1.671875* proc.memory.percent‚18.505167961120605 cpu‚25.138942475268 network.sent‚ 8.180493e+06 disk.sda1.out‚0 disk.rbd0.out‚1.125 disk.md0.in‚ 1.94140625# memory_percent‚96.0911512374878©(aœ•‚(:‘ ®±éÔÒÓk gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿#^\–‚(:Ñ ®±éÔг‹n disk.nvme0n1.in‚ 58.9921875 disk.md0.out‚ 443.1015625 disk.rbd2.in‚0) disk./.usagePercent‚83.73972752415135 disk.nvme1n1.in‚ 13.62109375 disk.sda.out‚460.99462890625 disk.rbd0.out‚1.125 disk.md0p1.out‚ 443.1015625 disk.rbd2.out‚1.125% disk./.usageGB‚2994.9382705688477 disk.dm-0.in‚ 72.61328125 cpu‚25.15098962826683! disk.nvme0n1.out‚ 27565.828125 disk.sdb2.in‚ 0.26953125 proc.cpu.threads‚17! disk.sdb2.out‚460.99462890625# disk.nvme1n1.out‚22181.42578125 disk.md0.in‚ 1.94140625 disk.sda.in‚1.671875 disk.sda1.in‚0 disk.rbd0.in‚0$ memory_percent‚95.78819274902344$ proc.memory.availableMB‚690.0625 disk.sda1.out‚0 disk.sdb1.out‚0 disk.md0p1.in‚ 1.94140625! disk.sda2.out‚460.99462890625) proc.memory.percent‚18.20361614227295 disk.sdb.out‚460.99462890625 network.sent‚ 8.189484e+06 network.recv‚ 1.44038e+06 disk.sdb1.in‚0 disk.sdb.in‚ 0.26953125 disk.sda2.in‚1.671875# proc.memory.rssMB‚ 2982.48046875 disk.dm-0.out‚49747.25390625ñ‘‰]’—‚(:‡ ½±éÔçÿm gpu.0.gpu‚51.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.9915 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TR£‘Z˜‚(:Ï ½±éÔÕƒ‘n disk.sdb2.in‚ 0.26953125! disk.sdb2.out‚463.14013671875 disk.rbd0.out‚1.125 disk.sdb1.in‚0 disk.sdb1.out‚0# disk.nvme1n1.out‚22209.63671875 disk.dm-0.in‚ 72.61328125 disk.sdb.out‚463.14013671875 disk.sda2.in‚1.671875" disk.nvme0n1.out‚ 27723.3046875& proc.memory.availableMB‚ 653.703125 proc.cpu.threads‚17 disk.sdb.in‚ 0.26953125 disk.nvme0n1.in‚ 58.9921875) proc.memory.percent‚18.42365264892578 disk.md0.out‚ 445.171875 disk.rbd2.in‚0 disk.sda.out‚463.14013671875 disk.sda1.in‚0 disk.rbd0.in‚0 cpu‚25.163602605977612 network.recv‚ 1.44368e+06 disk.rbd2.out‚1.125 disk.sda.in‚1.671875 disk.md0p1.in‚ 1.94140625 disk.dm-0.out‚49932.94140625 network.sent‚ 8.192538e+06 disk.sda1.out‚0 disk.md0p1.out‚ 445.171875! disk.sda2.out‚463.14013671875 disk.md0.in‚ 1.94140625$ memory_percent‚96.01011276245117 proc.memory.rssMB‚ 3018.53125( disk./.usagePercent‚83.7406301922347% disk./.usageGB‚2994.9705543518066 disk.nvme1n1.in‚ 13.62109375Ÿóo2™- _wandbruntime‚4224  _runtime‚4224Š+­OY*šÓT _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 4155333203456 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚44 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 4140478906011 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 167750417 % _timestamp‚1.7885001662908196e+09 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189767680 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32248 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1185 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1195376640 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚67535081 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 170502144 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚221 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚299662033867223 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 _step‚294 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 4156016242176 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20710 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 167729705 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚75859574 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚305633540895232 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚341551857861632 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚337206668088832 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13684416517 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 167729705 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1231 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚75859572 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 683038720 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚45452220 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 4139797838567 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14401142784 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20712 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 14135737856 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚91890843 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20710 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 4345360274944 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚615 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 14135737856 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20712 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚305633540895232 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1268 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 698106804 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1187728389 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚309789575140352 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚305633558898176 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 681067444 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1185 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 167750417 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3725 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 13002869760 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 683038720 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 194442240 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚75859574 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚223 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 167750417 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3725 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 13002869760 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 683038720 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 194442240 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚75859574 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚223 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 4140478906011 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 167750417 % _timestamp‚1.7885001662908196e+09 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189767680 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32248 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1185 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1195376640 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚67535081 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 170502144 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚221 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚299662033867223 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 _step‚294 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 4156016242176 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20710 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 167729705 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚75859574 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚305633540895232 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚341551857861632 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚337206668088832 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13684416517 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 167729705 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1231 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚75859572 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 683038720 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚45452220¦Ó–8¢2œ- _wandbruntime‚4225  _runtime‚4225ý€€c ÜR P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 14135737856 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚91899657 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚45461782 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1195376640 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 167774707 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 4345679623680 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚570 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚305687819805184 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20712 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚295572679173200 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32248 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚91878947 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 _step‚295 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚615 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14401142784 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135737856 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 701041664 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 4155823054336 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 4140967116895 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 681067444 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚309843624856576 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚341608809708544 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚337262959582720 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 170502144 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1187728389 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570ø g)žáR D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32248 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32005 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚341608615266304 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 4345850125824 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚91878947 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚299713646290095 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20712 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32005 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚305687801802240 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚224 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚22088258 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 701041664 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚45461780 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32248 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚91878947 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 _step‚295 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚615 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14401142784 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135737856 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 701041664 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 4155823054336 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 4140967116895 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 681067444 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚309843624856576 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚341608809708544 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚337262959582720 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 170502144 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1187728389 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 14135737856 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚91899657 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚45461782 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1195376640 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 167774707 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 4345679623680 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚570 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚305687819805184 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20712 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚295572679173200 B&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9880181550979614 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14300447702407837 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.36309754848480225 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2195594310760498 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9925368428230286 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.026280324906110764 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_3‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14012160897254944 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4748113751411438 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 epoch‚2 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_2_of_4‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 *"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 *"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5 8"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.539734959602356 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16123662889003754 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 7validation_average_jet_accuracy‚0.42678406834602356 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.991365373134613 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.0772344321012497 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15330305695533752 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.00537066487595439 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.38452380895614624 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9824561476707458 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4617730379104614 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03160824626684189 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21908243000507355 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2916666567325592 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0855964720249176 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.23850573599338531 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9918239116668701 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.07814982533454895 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.16152264177799225 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5336703062057495 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_6‚0 jet/accuracy_5_of_7‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07894736528396606 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9912824630737305 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.36973637342453003 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2038138210773468 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.49108096957206726 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9973883032798767 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14133690297603607 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9051460027694702 jet/accuracy_7_of_7‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1199839785695076 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.24727723002433777 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.990713894367218 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.8696330785751343 jet/accuracy_3_of_7‚0 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3607100546360016 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9900425672531128 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1606215238571167 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.39287617802619934 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4927855432033539 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0565476194024086 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_6‚0.6602563858032227 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26398831605911255 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8073872327804565 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 _runtime‚ 4240.94273844 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5097701549530029 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4791180491447449 particle/accuracy_2_of_5‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.49796876311302185 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3866012692451477 trainer/global_step‚3626 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994527101516724 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.16762451827526093 particle/accuracy_1_of_1‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 jet/accuracy_4_of_7‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5315223932266235 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.32955822348594666 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16666151583194733 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_3_of_5‚0.051920127123594284 (particle/accuracy‚0.8505755662918091 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5419642925262451 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2916666567325592 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.020378906279802322 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.45069441199302673 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9976115822792053 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 6 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.385185182094574 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12878787517547607 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2826588749885559 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7196969389915466 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4626302421092987 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994527101516724 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/2FªÛ¾;ÿ`Rt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4349019527435303 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14439718425273895 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.029753033071756363 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5545803904533386 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5284819602966309 )jet/accuracy_1_of_1‚0.476276695728302 ,jet/accuracy_4_of_5‚0.004704301245510578 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_6‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23969702422618866 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4665410816669464 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.625 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7381331324577332 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9940325617790222 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02430555783212185 8"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.535493791103363 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18041656911373138 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5153414011001587 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0030303029343485832 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2172619104385376 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.010998879559338093 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_6‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4542281925678253 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9811320900917053 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14763350784778595 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9963585734367371 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2169138640165329 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.49079862236976624 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4014471769332886 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21014657616615295 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06401979178190231 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4797821640968323 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1229906752705574 +particle/specificity‚0.9482867121696472 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2361111044883728 jet/accuracy_2_of_7‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2846801280975342 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.553697407245636 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.984195351600647 particle/accuracy_1_of_7‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4120640754699707 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_4_of_6‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4927855432033539 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5189374089241028 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2666667103767395 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.48514464497566223 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24406908452510834 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9923089146614075 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.34090909361839294 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_4_of_4‚0.00766606954857707 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1468254178762436 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4895833432674408 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 +jet/accuracy_3_of_4‚0.06732451170682907 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_7‚1 particle/accuracy_6_of_7‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4911511540412903 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5133333802223206 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.25970888137817383 particle/accuracy_4_of_4‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33225739002227783 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975091814994812 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.19999998807907104 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3759690225124359 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1924242377281189 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_4‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3676406443119049 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2096889764070511 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.48956823348999023 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9911787509918213 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9833333492279053 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3795207142829895 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 )loss/val_total_loss‚44.12614850288044 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16369763016700745 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13485556840896606 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18963605165481567 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4561743438243866 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4809872806072235 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5209288001060486 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4166666865348816 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21646790206432343 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24343979358673096 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.34854376316070557 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.015750816091895103 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3743664622306824 particle/accuracy_5_of_5‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_7_of_7‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.007823129184544086 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_6‚1 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 8"Purity/*FRt2SRqqt*FBt/event_purity‚0.026615085080266 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5043665170669556 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.47165748476982117 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.28618618845939636 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.30000001192092896 jet/accuracy_5_of_6‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 6 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.991912305355072 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9629628658294678 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.37037035822868347 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.47868841886520386 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9862231612205505 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5011674165725708 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2306881695985794 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17622806131839752 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1568944752216339 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_7‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996220469474792 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 +jet/accuracy_2_of_5‚0.40974780917167664 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.21350574493408203 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9893864393234253 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4859520494937897 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.18851351737976074 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.09961264580488205 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 +"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.75 *jet/accuracy_1_of_4‚0.9090468287467957 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15224774181842804 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3971172273159027 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 particle/accuracy_1_of_2‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40123459696769714 _step‚296 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.549842357635498 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.023872511461377144 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3534225821495056 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1951236128807068 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5147930979728699 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4961262047290802 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37125954031944275 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9913617968559265 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_2‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4183292090892792 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 jet/accuracy_1_of_7‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9880674481391907 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2916666567325592 *jet/accuracy_1_of_6‚0.9807692170143127 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17492146790027618 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9290779829025269 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 +jet/accuracy_2_of_3‚0.23017491400241852 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4232456386089325 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1785714328289032 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 5 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.98902428150177 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 particle/accuracy_4_of_5‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_6_of_6‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5034101605415344 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15402285754680634 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 jet/accuracy_6_of_7‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0793650820851326 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15783698856830597 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.44820651412010193 ,jet/accuracy_3_of_3‚0.020790990442037582 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833134651184 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4652777910232544 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4285714328289032 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6260824203491211 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15039239823818207 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0014773586299270391 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6926787495613098 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 -"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4375 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5015084743499756 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074074625968933 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.00657894741743803 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.11777778714895248 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24701794981956482 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9848693609237671 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08923260122537613 +jet/accuracy_3_of_6‚0.12820513546466827 particle/accuracy_2_of_7‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5159338116645813 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4711270332336426 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21042156219482422 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3705897629261017 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.39549803733825684 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5025690197944641 % _timestamp‚1.7885001831622114e+09 5 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.43271604180336 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0666923075914383 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0017636684933677316 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4954860210418701 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4730002284049988 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9962422251701355 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3672236204147339 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666567325592 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2430555522441864 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_6‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03879310190677643 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 jet/accuracy_5_of_5‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2916666567325592 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14300447702407837 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 +particle/sensitivity‚0.3225739002227783 'particle/f_score‚0.4024673402309418 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 <"Purity/2FRt*SRqqt2FBt/event_purity‚0.0058139534667134285 particle/accuracy_3_of_4‚1 particle/accuracy_1_of_5‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.4899961054325104 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4657973647117615 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7857142686843872 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ;"Purity/*FRt*SRqqt3FBt/event_purity‚0.056737590581178665 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9778911471366882 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.12666024267673492{°£”ó£ü¥ & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33225739002227783 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975091814994812 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.19999998807907104 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3759690225124359 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1924242377281189 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_4‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3676406443119049 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2096889764070511 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.48956823348999023 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9911787509918213 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9833333492279053 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3795207142829895 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 )loss/val_total_loss‚44.12614850288044 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16369763016700745 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13485556840896606 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18963605165481567 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4561743438243866 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4809872806072235 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5209288001060486 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4166666865348816 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21646790206432343 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24343979358673096 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.34854376316070557 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.015750816091895103 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3743664622306824 particle/accuracy_5_of_5‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_7_of_7‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.007823129184544086 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_6‚1 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 8"Purity/*FRt2SRqqt*FBt/event_purity‚0.026615085080266 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5043665170669556 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.47165748476982117 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.28618618845939636 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.30000001192092896 jet/accuracy_5_of_6‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 6 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.991912305355072 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9629628658294678 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.37037035822868347 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.47868841886520386 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9862231612205505 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5011674165725708 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2306881695985794 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17622806131839752 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1568944752216339 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_7‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996220469474792 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 +jet/accuracy_2_of_5‚0.40974780917167664 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.21350574493408203 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9893864393234253 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4859520494937897 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.18851351737976074 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.09961264580488205 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 ("Purity/3FRt3SRqqt*FBt/event_purity‚³¹Ît0 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 +"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.75 *jet/accuracy_1_of_4‚0.9090468287467957 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15224774181842804 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3971172273159027 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 particle/accuracy_1_of_2‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40123459696769714 _step‚296 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.549842357635498 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.023872511461377144 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3534225821495056 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1951236128807068 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5147930979728699 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4961262047290802 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37125954031944275 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9913617968559265 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_2‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4183292090892792 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 jet/accuracy_1_of_7‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9880674481391907 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2916666567325592 *jet/accuracy_1_of_6‚0.9807692170143127 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17492146790027618 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9290779829025269 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 +jet/accuracy_2_of_3‚0.23017491400241852 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4232456386089325 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1785714328289032 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 5 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.98902428150177 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 particle/accuracy_4_of_5‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_6_of_6‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5034101605415344 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15402285754680634 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 jet/accuracy_6_of_7‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0793650820851326 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15783698856830597 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.44820651412010193 ,jet/accuracy_3_of_3‚0.020790990442037582 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833134651184 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4652777910232544 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4285714328289032 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6260824203491211 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15039239823818207 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0014773586299270391 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6926787495613098 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 -"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4375 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5015084743499756 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074074625968933 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.00657894741743803 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.11777778714895248 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24701794981956482 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9848693609237671 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08923260122537613 +jet/accuracy_3_of_6‚0.12820513546466827 particle/accuracy_2_of_7‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5159338116645813 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4711270332336426 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21042156219482422 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3705897629261017 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.39549803733825684 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5025690197944641 % _timestamp‚1.7885001831622114e+09 5 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.43271604180336 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0666923075914383 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0017636684933677316 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4954860210418701 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4730002284049988 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9962422251701355 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3672236204147339 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666567325592 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2430555522441864 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_6‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03879310190677643 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 jet/accuracy_5_of_5‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2916666567325592 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14300447702407837 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 +particle/sensitivity‚0.3225739002227783 'particle/f_score‚0.4024673402309418 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 <"Purity/2FRt*SRqqt2FBt/event_purity‚0.0058139534667134285 particle/accuracy_3_of_4‚1 particle/accuracy_1_of_5‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.4899961054325104 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4657973647117615 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7857142686843872 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ;"Purity/*FRt*SRqqt3FBt/event_purity‚0.056737590581178665 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9778911471366882 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.12666024267673492 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5646551847457886 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22685186564922333 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.48310723900794983 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4440050721168518 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5614035129547119 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9953519105911255 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01954638957977295 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9880181550979614 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14300447702407837 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.36309754848480225 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2195594310760498 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9925368428230286 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.026280324906110764 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_3‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14012160897254944 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4748113751411438 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 epoch‚2 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_2_of_4‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 *"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 *"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5 8"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.539734959602356 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16123662889003754 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 7validation_average_jet_accuracy‚0.42678406834602356 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.991365373134613 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.0772344321012497 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15330305695533752 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.00537066487595439 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.38452380895614624 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9824561476707458 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4617730379104614 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03160824626684189 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21908243000507355 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2916666567325592 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0855964720249176 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.23850573599338531 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9918239116668701 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.07814982533454895 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.16152264177799225 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5336703062057495 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_6‚0 jet/accuracy_5_of_7‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07894736528396606 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9912824630737305 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.36973637342453003 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2038138210773468 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.49108096957206726 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9973883032798767 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14133690297603607 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9051460027694702 jet/accuracy_7_of_7‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1199839785695076 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.24727723002433777 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.990713894367218 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.8696330785751343 jet/accuracy_3_of_7‚0 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3607100546360016 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9900425672531128 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1606215238571167 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.39287617802619934 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4927855432033539 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0565476194024086 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_6‚0.6602563858032227 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26398831605911255 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8073872327804565 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 _runtime‚ 4240.94273844 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5097701549530029 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4791180491447449 particle/accuracy_2_of_5‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.49796876311302185 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3866012692451477 trainer/global_step‚3626 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994527101516724 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.16762451827526093 particle/accuracy_1_of_1‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 jet/accuracy_4_of_7‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5315223932266235 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.32955822348594666 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16666151583194733 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_3_of_5‚0.051920127123594284 (particle/accuracy‚0.8505755662918091 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5419642925262451 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2916666567325592 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.020378906279802322 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.45069441199302673 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9976115822792053 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 6 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.385185182094574 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12878787517547607 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2826588749885559 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7196969389915466 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4626302421092987 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994527101516724 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4349019527435303 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14439718425273895 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.029753033071756363 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5545803904533386 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5284819602966309 )jet/accuracy_1_of_1‚0.476276695728302 ,jet/accuracy_4_of_5‚0.004704301245510578 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_6‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23969702422618866 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4665410816669464 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.625 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7381331324577332 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9940325617790222 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02430555783212185 8"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.535493791103363 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18041656911373138 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5153414011001587 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0030303029343485832 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2172619104385376 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.010998879559338093 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_6‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4542281925678253 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9811320900917053 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14763350784778595 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9963585734367371 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2169138640165329 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.49079862236976624 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4014471769332886 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21014657616615295 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06401979178190231 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4797821640968323 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1229906752705574 +particle/specificity‚0.9482867121696472 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2361111044883728 jet/accuracy_2_of_7‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2846801280975342 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.553697407245636 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.984195351600647 particle/accuracy_1_of_7‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4120640754699707 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_4_of_6‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4927855432033539 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5189374089241028 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2666667103767395 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.48514464497566223 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24406908452510834 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9923089146614075 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.34090909361839294 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_4_of_4‚0.00766606954857707 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1468254178762436 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4895833432674408 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 +jet/accuracy_3_of_4‚0.06732451170682907 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_7‚1 particle/accuracy_6_of_7‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4911511540412903 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5133333802223206 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.25970888137817383 particle/accuracy_4_of_4‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0¨Sf,I¤‚(:… Û±éÔ¡Îm gpu.0.gpu‚5.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚32.0 gpu.0.powerWatts‚86.502' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚21.6255 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cº>_¥‚(:Ô Û±éÔ—Åm! disk.sdb2.out‚466.06298828125) proc.memory.availableMB‚ 1120.77734375# proc.memory.rssMB‚ 3207.94140625 disk.sda2.in‚1.671875 disk.rbd0.out‚1.125 disk.dm-0.in‚ 72.61328125! disk.sda2.out‚466.06298828125 disk.nvme0n1.in‚ 58.9921875 disk.md0.in‚ 1.94140625 disk.rbd2.in‚0 disk.sda.in‚1.671875 disk.md0p1.in‚ 1.94140625 disk.nvme0n1.out‚ 27933.28125 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚93.15931797027588 disk.md0.out‚ 447.984375 disk.rbd2.out‚1.125 disk.sdb2.in‚ 0.26953125 disk.rbd0.in‚0 network.sent‚ 8.304747e+06 network.recv‚ 1.453378e+06% disk./.usageGB‚2996.3327674865723 disk.sdb.out‚466.06298828125 disk.sdb1.in‚0* proc.memory.percent‚19.579720497131348 proc.cpu.threads‚17 disk.nvme1n1.in‚ 13.62109375" disk.nvme1n1.out‚ 22390.8515625 disk.sda.out‚466.06298828125 disk.sdb1.out‚0 disk.md0p1.out‚ 447.984375 cpu‚25.188570953498278) disk./.usagePercent‚83.77871824161323 disk.dm-0.out‚ 50324.1328125 disk.sdb.in‚ 0.26953125AÖÅ“¦‚(:ˆ ê±éÔ†ý¹m gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚124.661' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.16525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ‘|‰K§‚(:× ê±éÔ©£÷n network.recv‚ 1.45901e+06 disk.sdb1.out‚0" disk.nvme0n1.out‚ 29555.5078125) proc.memory.percent‚18.38083267211914) disk./.usagePercent‚83.78563635038718 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 72.61328125 disk.rbd2.in‚0$ memory_percent‚92.77598857879639 proc.cpu.threads‚17 disk.sda.out‚467.93701171875 disk.rbd2.out‚1.125 disk.sdb2.in‚ 0.26953125 network.sent‚ 8.427972e+06 disk.sda.in‚1.671875 disk.sdb1.in‚0 disk.md0p1.out‚ 449.80078125 disk.sdb.in‚ 0.26953125# disk.nvme1n1.out‚22468.48828125) proc.memory.availableMB‚ 1183.58203125 cpu‚25.188068976904294 disk.md0.in‚ 1.94140625 disk.md0.out‚ 449.80078125$ disk./.usageGB‚2996.580192565918 disk.rbd0.out‚1.125 disk.sdb.out‚467.93701171875 disk.sda2.in‚1.671875! disk.sdb2.out‚467.93701171875! proc.memory.rssMB‚ 3011.515625! disk.sda2.out‚467.93701171875 disk.nvme1n1.in‚ 13.62109375 disk.sda1.out‚0 disk.md0p1.in‚ 1.94140625 disk.dm-0.out‚52023.99609375 disk.Z(>hnvme0n1.in‚ 58.9921875åòo,2¨- _wandbruntime‚4269  _runtime‚4269ó­jwÇ©Á & lr-AdamW/pg1‚0.0009270952765083373 & lr-AdamW/pg2‚0.0009270952765083373 trainer/global_step‚3649 % _timestamp‚1.7885002121087806e+09 _runtime‚4269.886105467 _step‚297*j>ÌªÆ & lr-AdamW/pg2‚0.0009270952765083373 trainer/global_step‚3649 % _timestamp‚1.7885002121087806e+09 _runtime‚4269.886105467 _step‚297 & lr-AdamW/pg1‚0.0009270952765083373©‘…öf2«- _wandbruntime‚4269  _runtime‚4269Žvò'¬ìO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22283177 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚312376389371392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4187557886976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22283813 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393472 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76519758 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚302138509286452 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚312375461396992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚308188813481472 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚302137584272088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92538992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚308188813481472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76519756 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4170995690888 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169058748 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76519758 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76519756 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92538992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚308188831484416 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4186647915520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚297966588581200 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45851518 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169085094 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚339979379339776 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4186647915520 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45851516 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚344357583306240 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚308188831484416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚339979403279872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚638 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68134693 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7885002121100216e+09 _runtime‚4269.887470494 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92565336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚312375461396992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚308188813481472 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚302137584272088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92538992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚308188813481472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76519756 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4170995690888 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169058748 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76519758 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76519756 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92538992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚308188831484416 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4186647915520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚297966588581200 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45851518 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169085094 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚339979379339776 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4186647915520 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45851516 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚344357583306240 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚308188831484416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚339979403279872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249ª‚ûü¾2®- _wandbruntime‚4270  _runtime‚4270ê, žü&¯öM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92590193 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚308270432567296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚308270450570240 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4187769375744 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169103739 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92563823 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4188679360512 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169103739 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45863581 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4379110284288 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4188679360512 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚340069628984320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76539918 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92590193 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68153185 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45863583 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 _runtime‚4270.958027694 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169130111 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚302218644154482 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚302217719139910 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚298045606175744 trainer/global_step‚3649 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚312459129930752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22289602 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚643 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚340069605044224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76539918 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68152540 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ÚX•°ûM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22288959 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚344448976425984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚308270432567296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4379347441664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚302218644154482 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚302217719139910 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚298045606175744 trainer/global_step‚3649 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚312459129930752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22289602 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚643 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚340069605044224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76539918 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68152540 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92590193 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚308270432567296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚308270450570240 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4187769375744 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169103739 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92563823 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4188679360512 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169103739 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45863581 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4379110284288 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4188679360512 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚340069628984320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76539918 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92590193 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68153185 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45863583 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 _runtime‚4270.958027694 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169130111 F¢ü“¸‚(:ˆ —²éÔ…øïk gpu.0.gpu‚14.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚182.031' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.50775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹µ AV¹‚(:Ë —²éÔŸºël disk.sda2.in‚1.671875 disk.sda2.out‚ 472.662109375 disk.md0.out‚ 454.36328125 disk.sda.in‚1.671875 disk.sda.out‚ 472.662109375 network.sent‚ 8.546696e+06 network.recv‚ 1.472138e+06% disk./.usageGB‚2996.0529136657715 disk.rbd0.out‚1.125 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.md0p1.out‚ 454.36328125 disk.sda1.in‚0 disk.sdb.out‚ 472.662109375 disk.rbd2.out‚1.125! proc.memory.rssMB‚ 3011.515625 disk.dm-0.in‚ 72.61328125 disk.nvme0n1.in‚ 58.9921875) proc.memory.percent‚18.38083267211914 cpu‚25.186197650464006 proc.cpu.threads‚17 disk.rbd0.in‚0# disk.nvme1n1.out‚22723.78515625 disk.sdb2.out‚ 472.662109375$ memory_percent‚95.97830772399902 disk.dm-0.out‚52751.66015625 disk.sdb.in‚ 0.26953125 disk.nvme1n1.in‚ 13.62109375 disk.md0.in‚ 1.94140625 disk.sdb2.in‚ 0.26953125) disk./.usagePercent‚83.77089341165572 disk.sda1.out‚0 disk.nvme0n1.out‚ 30027.875' proc.memory.availableMB‚ 658.9140625 disk.md0p1.in‚ 1.94140625ÿ*Xo“º‚(:ˆ ¦²éÔêôél gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚176.129' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.03225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z­V»‚(:Ë ¦²éÔìm) disk./.usagePercent‚83.74016248580135$ disk./.usageGB‚2994.953826904297 disk.sda2.in‚1.671875 disk.sdb2.out‚ 474.326171875' proc.memory.availableMB‚ 659.0078125) proc.memory.percent‚18.38083267211914 disk.sda2.out‚ 474.326171875 disk.rbd2.out‚1.125 disk.sda1.out‚0 disk.sdb1.in‚0 disk.dm-0.out‚52884.10546875$ memory_percent‚95.97773551940918 disk.sdb.in‚ 0.26953125 disk.sdb2.in‚ 0.26953125 disk.rbd0.in‚0 disk.rbd0.out‚1.125 disk.dm-0.in‚ 72.61328125 disk.sdb.out‚ 474.326171875 disk.nvme0n1.in‚ 58.9921875 disk.md0.out‚ 455.96484375 disk.rbd2.in‚0 disk.sda1.in‚0! proc.memory.rssMB‚ 3011.515625 cpu‚25.18566516179167 proc.cpu.threads‚17 disk.md0p1.in‚ 1.94140625 disk.nvme1n1.in‚ 13.62109375# disk.nvme1n1.out‚22767.94921875 disk.sdb1.out‚0 disk.md0p1.out‚ 455.96484375 network.recv‚ 1.476755e+06 disk.nvme0n1.out‚ 30116.15625 disk.md0.in‚ 1.94140625 disk.sda.out‚ 474.326171875 network.sent‚ 8.555184e+06 disk.sda.in‚1.671875œ32¼- _wandbruntime‚4323  _runtime‚4323 ÉƽÀ & lr-AdamW/pg2‚0.0009221342193510007 trainer/global_step‚3699 % _timestamp‚1.7885002655955334e+09 _runtime‚ 4323.37274577 _step‚301 & lr-AdamW/pg1‚0.0009221342193510007|’-˾Š& lr-AdamW/pg1‚0.0009221342193510007 & lr-AdamW/pg2‚0.0009221342193510007 trainer/global_step‚3699 % _timestamp‚1.7885002655955334e+09 _runtime‚ 4323.37274577 _step‚301­ iy2¿- _wandbruntime‚4323  _runtime‚4323ûÏ$ò'ÀìO OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4436230401024 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69025605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22570234 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93783894 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4436446600192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚316513431087616 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚316512503113216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚312269767772672 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚312269767772672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46454766 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚344490664562176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚312269785775616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46454768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4242735340544 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚316513431087616 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4243645312000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚348927135102464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77527756 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚348926894963200 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚306144341254396 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77527758 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0}Í ¦¡ÁñO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4242735340544 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚316513431087616 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4243645312000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚348927135102464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77527756 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚348926894963200 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚306144341254396 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77527758 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4436230401024 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69025605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22570234 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93783894 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4436446600192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚316513431087616 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚316512503113216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚312269767772672 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚312269767772672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46454766 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚344490664562176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚312269785775616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46454768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144®H·´2Â- _wandbruntime‚4324  _runtime‚4324¬\˜ü&ÃöM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93808725 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4243856800768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 _step‚303 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4244766785536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69042808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚344580914206720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4437615195136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235060224 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚316595243659264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚349018270401536 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 % _timestamp‚1.7885002665431893e+09 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46466831 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4227990219274 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171356641 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚599 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚306224476122218 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚301996502942304 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚312351404861440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚344580890266624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚312351386858496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22575977 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚349018529401856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 171383013 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77547916 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93835095 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚312351404861440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚316596171646976 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚306225401136790 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4244766785536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46466833 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77547918 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69043407 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 171356641 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171383013 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77547918 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22576574 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615JB×'ÄûM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 171383013 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77547916 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93835095 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚312351404861440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚316596171646976 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚306225401136790 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4244766785536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46466833 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77547918 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69043407 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 171356641 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171383013 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77547918 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22576574 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93808725 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4243856800768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 _step‚303 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4244766785536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69042808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚344580914206720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4437615195136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235060224 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚316595243659264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚349018270401536 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 % _timestamp‚1.7885002665431893e+09 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46466831 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4227990219274 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171356641 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚599 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚306224476122218 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚301996502942304 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚312351404861440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚344580890266624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚312351386858496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22575977 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚349018529401856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784¯³vÞ2Å- _wandbruntime‚4324  _runtime‚4324QZ2Ó\ÆÖ 1loss/FBt4/assignment_loss‚0.37836318861125273 /loss/FRt2/detection_loss‚2.3658439564030793 /loss/FRt4/detection_loss‚2.9741157432397207 % _timestamp‚1.7885002665451267e+09 /loss/FRt4/assignment_loss‚3.278690840800603 2loss/SRqqt3/assignment_loss‚0.9976153702095703 /loss/FRt3/detection_loss‚1.6100065136357906 0loss/SRqqt1/detection_loss‚2.243357403079669 1loss/SRqqt2/detection_loss‚2.9479810846479317 0loss/SRqqt4/detection_loss‚2.209112899005413 trainer/global_step‚3699 _step‚304 0loss/FRt3/assignment_loss‚1.8591263899727473 2loss/SRqqt1/assignment_loss‚1.5083673681054885 1loss/SRqqt2/assignment_loss‚1.929665838417254 _runtime‚4324.322092457 2loss/SRqqt4/assignment_loss‚1.3258425634354354 /loss/FBt3/detection_loss‚2.4329695850610733 1loss/SRqqt3/detection_loss‚0.9726748203306839 /loss/FBt1/detection_loss‚0.8136084955592181 /loss/FBt2/detection_loss‚2.5322168370087943 .loss/FBt4/detection_loss‚2.677346248002279 0loss/FRt1/assignment_loss‚2.5407983093726925 1loss/FBt3/assignment_loss‚0.19298551722020818 %loss/total_loss‚44.08685415494821 0loss/FRt2/assignment_loss‚3.9118835230668387 .loss/FRt1/detection_loss‚1.999345981322865 epoch‚3 1loss/FBt1/assignment_loss‚0.15658315482462043 0loss/FBt2/assignment_loss‚0.2283525236149823FqOaÇÛ 0loss/FRt2/assignment_loss‚3.9118835230668387 .loss/FRt1/detection_loss‚1.999345981322865 epoch‚3 1loss/FBt1/assignment_loss‚0.15658315482462043 0loss/FBt2/assignment_loss‚0.2283525236149823 1loss/FBt4/assignment_loss‚0.37836318861125273 /loss/FRt2/detection_loss‚2.3658439564030793 /loss/FRt4/detection_loss‚2.9741157432397207 % _timestamp‚1.7885002665451267e+09 /loss/FRt4/assignment_loss‚3.278690840800603 2loss/SRqqt3/assignment_loss‚0.9976153702095703 /loss/FRt3/detection_loss‚1.6100065136357906 0loss/SRqqt1/detection_loss‚2.243357403079669 1loss/SRqqt2/detection_loss‚2.9479810846479317 0loss/SRqqt4/detection_loss‚2.209112899005413 trainer/global_step‚3699 _step‚304 0loss/FRt3/assignment_loss‚1.8591263899727473 2loss/SRqqt1/assignment_loss‚1.5083673681054885 1loss/SRqqt2/assignment_loss‚1.929665838417254 _runtime‚4324.322092457 2loss/SRqqt4/assignment_loss‚1.3258425634354354 /loss/FBt3/detection_loss‚2.4329695850610733 1loss/SRqqt3/detection_loss‚0.9726748203306839 /loss/FBt1/detection_loss‚0.8136084955592181 /loss/FBt2/detection_loss‚2.5322168370087943 .loss/FBt4/detection_loss‚2.677346248002279 0loss/FRt1/assignment_loss‚2.5407983093726925 1loss/FBt3/assignment_loss‚0.19298551722020818 %loss/total_loss‚44.08685415494821°2D´‘È‚(:† µ²éÔ¥Ó‚m gpu.0.gpu‚51.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚177.688' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.422 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨ègUÉ‚(:Ê µ²éÔþ‰ëm disk.md0p1.in‚ 1.94140625# disk.nvme0n1.out‚30188.20703125 disk.md0.in‚ 1.94140625) disk./.usagePercent‚83.74212535957398 disk.sdb2.in‚ 0.26953125 disk.sda.in‚1.671875 disk.md0p1.out‚457.3125) proc.memory.percent‚18.38085651397705 disk.md0.out‚457.3125 disk.sdb2.out‚475.7275390625 disk.dm-0.in‚ 72.61328125 disk.dm-0.out‚52956.35546875 disk.nvme0n1.in‚ 58.9921875 disk.nvme1n1.in‚ 13.62109375 disk.rbd2.out‚1.125& proc.memory.availableMB‚ 658.484375 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚1.125 disk.sda2.in‚1.671875 disk.sda2.out‚475.7275390625$ disk./.usageGB‚2995.024028778076 disk.sdb.out‚475.7275390625 disk.rbd2.in‚0$ memory_percent‚95.98093032836914 cpu‚25.18496179290486 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.sdb1.out‚0" disk.nvme1n1.out‚ 22768.1484375 network.recv‚ 1.479926e+06 disk.sda.out‚475.7275390625 disk.sdb.in‚ 0.26953125 network.sent‚ 8.571081e+06 disk.sda1.in‚0# proc.memory.rssMB‚ 3011.51953125s«¶¶‘Ê‚(:† IJéÔ€²ìl gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚124.044' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.011 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D¨ÈXË‚(:Í IJéÔ¾Ö»m disk.sdb2.out‚ 477.361328125 disk.dm-0.in‚ 72.61328125 disk.rbd0.out‚1.125 disk.sdb1.in‚0 disk.md0p1.out‚ 458.88671875 disk.rbd2.in‚0 disk.rbd2.out‚1.125 proc.cpu.threads‚17 network.sent‚ 8.667876e+06 network.recv‚ 1.485301e+06 disk.sda2.out‚ 477.361328125# disk.nvme0n1.out‚30216.30078125# disk.nvme1n1.out‚22768.26171875# proc.memory.rssMB‚ 3011.51953125 disk.rbd0.in‚0 disk.md0p1.in‚ 1.94140625 disk.dm-0.out‚ 52984.5625 disk.sdb.in‚ 0.26953125 disk.nvme0n1.in‚ 58.9921875 disk.md0.out‚ 458.88671875) disk./.usagePercent‚83.74393389555664$ disk./.usageGB‚2995.088710784912 disk.sda1.in‚0 disk.sdb1.out‚0 disk.nvme1n1.in‚ 13.62109375( proc.memory.availableMB‚ 658.48046875 cpu‚25.18444644779922 disk.sdb2.in‚ 0.26953125 disk.sda1.out‚0 disk.sda2.in‚1.671875$ memory_percent‚95.98095417022705 disk.sda.out‚ 477.361328125 disk.sdb.out‚ 477.361328125 disk.md0.in‚ 1.94140625) proc.memory.percent‚18.38085651397705 disk.sda.in‚1.671875Nê*"Ì‚(:… Ó²éÔÃÐæl gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚178.02' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.505 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dCãÒÍ‚(:Ó Ó²éÔŒ‘Ñm disk.sdb.out‚479.40576171875 disk.sda2.in‚1.671875 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd0.out‚1.125 disk.md0p1.out‚ 460.87109375# disk.nvme1n1.out‚22768.69140625 disk.rbd2.out‚1.125$ memory_percent‚95.97797393798828% proc.memory.availableMB‚ 658.96875) proc.memory.percent‚18.38085651397705 disk.sdb1.in‚0# disk.nvme0n1.out‚30320.83984375 disk.sda.out‚479.40576171875 disk.rbd2.in‚0 network.sent‚ 8.673274e+06$ disk./.usageGB‚2995.185405731201 disk.nvme1n1.in‚ 13.62109375 disk.md0.out‚ 460.87109375) disk./.usagePercent‚83.74663752672487 disk.dm-0.in‚ 72.6171875! disk.sda2.out‚479.40576171875 disk.sda.in‚1.671875# proc.memory.rssMB‚ 3011.51953125 disk.dm-0.out‚ 53089.53125 disk.sdb.in‚ 0.26953125! disk.sdb2.out‚479.40576171875 disk.sdb2.in‚ 0.26953125 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.nvme0n1.in‚ 58.9960™ã¶Œ9375 disk.md0.in‚ 1.94140625 cpu‚25.18381423654769 network.recv‚ 1.488996e+06 disk.sda1.in‚0 disk.md0p1.in‚ 1.94140625i”2Î- _wandbruntime‚4375  _runtime‚4375jȾÇÏÁ % _timestamp‚1.7885003175260878e+09 _runtime‚4375.303360214 _step‚305 & lr-AdamW/pg1‚0.0009170906452750526 & lr-AdamW/pg2‚0.0009170906452750526 trainer/global_step‚3749¡E\‘ÌÐÆ & lr-AdamW/pg1‚0.0009170906452750526 & lr-AdamW/pg2‚0.0009170906452750526 trainer/global_step‚3749 % _timestamp‚1.7885003175260878e+09 _runtime‚4375.303360214 _step‚305±ñd¦52Ñ- _wandbruntime‚4375  _runtime‚4375€Íoï'ÒéO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4298822765568 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚320649544829440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3749 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78535756 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69916204 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4494713991680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚310152023251068 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚305868348035600 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95055140 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚320650472803840 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚353496916498432 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚353496663776256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚ 4375.3044744 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95028796 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78535758 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4298822765568 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4494942773760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22857568 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22858186 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4299732737024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚310151098236704 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78535756 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4283658176108 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78535758 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4282750201104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _step‚306 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4299732737024 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78535758 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4282750201104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _step‚306 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4299732737024 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚320649544829440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3749 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78535756 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69916204 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4494713991680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚310152023251068 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚305868348035600 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95055140 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚320650472803840 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚353496916498432 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚353496663776256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚ 4375.3044744 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95028796 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78535758 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4298822765568 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4494942773760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22857568 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22858186 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4299732737024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚310151098236704 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78535756 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4283658176108 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4300854210560 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚310232158119098 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚305947365630144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95079997 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚305947382669504 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚316432359152640 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 trainer/global_step‚3749 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95053627 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47070081 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚353588306680320 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69933324 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22863243 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4300854210560 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚316432359152640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95079997 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4283867474382 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4495882676736 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 % _timestamp‚1.7885003184813552e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78555918 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69933942 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚320733213363200 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚320733213363200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22863859 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚349092199429120 _runtime‚4376.259207516 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95053627 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47070083 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4284775449594 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4299944225792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173635915 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚310231233104526 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78555916 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4496107251200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173635915 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚320732285375488 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144JÔF&ÖûM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚316432341149696 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69933324 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22863243 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4300854210560 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚316432359152640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95079997 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4283867474382 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4495882676736 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 % _timestamp‚1.7885003184813552e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78555918 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69933942 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚320733213363200 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚320733213363200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22863859 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚349092199429120 _runtime‚4376.259207516 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95053627 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47070083 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4284775449594 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4299944225792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173635915 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚310231233104526 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78555916 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4496107251200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173635915 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚320732285375488 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78555918 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173609543 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78555916 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4300854210560 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚310232158119098 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚305947365630144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95079997 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚305947382669504 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚316432359152640 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 trainer/global_step‚3749 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95053627 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47070081 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚353588306680320 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚320431694358016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96273698 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96300042 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79543758 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚358066411692032 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚309819227762800 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4355820162048 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79543758 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23144768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70805408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47661268 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 trainer/global_step‚3799 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4354910190592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175843800 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚353513258947072 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4552919768576 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚309819244802160 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚320431676355072 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚358066154775552 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4553152744960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚324786586545664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚324787514520064 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79543758 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23144768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70805408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47661268 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 trainer/global_step‚3799 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4354910190592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175843800 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚353513258947072 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4552919768576 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚309819244802160 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚320431676355072 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚358066154775552 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4553152744960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚324786586545664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚324787514520064 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚320431694358016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96273698 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96300042 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79543758 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚358066411692032 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚309819227762800 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4355820162048 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚324869327091712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚320513295440896 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4356031650816 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175888817 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚607 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79563918 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47673331 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚358157801203200 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175862445 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚320513313443840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23150535 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79563916 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96324899 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79563918 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70823261 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4356941635584 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175862445 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚353603460711424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4554316551680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚309898262396704 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚314237990086834 trainer/global_step‚3799 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚324870255079424 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4554087782912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4339744729490 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23150535 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79563916 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96324899 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79563918 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70823261 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4356941635584 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175862445 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚353603460711424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4554316551680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚309898262396704 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚314237990086834 trainer/global_step‚3799 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚324870255079424 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4554087782912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4339744729490 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚324869327091712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚320513295440896 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4356031650816 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175888817 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚607 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79563918 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47673331 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚358157801203200·û{2ë- _wandbruntime‚4428  _runtime‚4428ÔŒ¨ÂYìÓ .loss/FRt4/detection_loss‚2.790892843405406 /loss/FBt4/detection_loss‚2.3678298936440396 %loss/total_loss‚43.46529520660986 0loss/FRt2/assignment_loss‚3.3905636497906277 0loss/FBt3/assignment_loss‚0.5881310667027719 1loss/SRqqt1/detection_loss‚2.0599127506667916 1loss/SRqqt2/detection_loss‚2.7672965793774047 % _timestamp‚1.7885003705754006e+09 _runtime‚4428.352402241 0loss/FRt3/assignment_loss‚1.7758292017074733 /loss/FRt1/detection_loss‚1.6945256156778972 .loss/FRt3/detection_loss‚2.057361745203917 _step‚312 /loss/FRt4/assignment_loss‚2.776418079932531 2loss/SRqqt1/assignment_loss‚1.8723105165091427 2loss/SRqqt3/assignment_loss‚1.0800924841985777 2loss/SRqqt4/assignment_loss‚1.8963399201631546 0loss/SRqqt4/detection_loss‚2.294087961741856 0loss/FBt2/assignment_loss‚0.1750656780121582 .loss/FRt2/detection_loss‚2.470711051708176 1loss/SRqqt3/detection_loss‚1.0754820404591057 .loss/FBt2/detection_loss‚2.650021391255515 0loss/FRt1/assignment_loss‚2.3542733629544577 1loss/SRqqt2/assignment_loss‚1.944341843594508 /loss/FBt1/detection_loss‚0.6461693874572392 trainer/global_step‚3799 1loss/FBt1/assignment_loss‚0.09564997408603244 0loss/FBt4/assignment_loss‚0.2104750525974081 /loss/FBt3/detection_loss‚2.4315131157636642 epoch‚3Ê·æ“^íØ 0loss/FBt4/assignment_loss‚0.2104750525974081 /loss/FBt3/detection_loss‚2.4315131157636642 epoch‚3 .loss/FRt4/detection_loss‚2.790892843405406 /loss/FBt4/detection_loss‚2.3678298936440396 %loss/total_loss‚43.46529520660986 0loss/FRt2/assignment_loss‚3.3905636497906277 0loss/FBt3/assignment_loss‚0.5881310667027719 1loss/SRqqt1/detection_loss‚2.0599127506667916 1loss/SRqqt2/detection_loss‚2.7672965793774047 % _timestamp‚1.7885003705754006e+09 _runtime‚4428.352402241 0loss/FRt3/assignment_loss‚1.7758292017074733 /loss/FRt1/detection_loss‚1.6945256156778972 .loss/FRt3/detection_loss‚2.057361745203917 _step‚312 /loss/FRt4/assignment_loss‚2.776418079932531 2loss/SRqqt1/assignment_loss‚1.8723105165091427 2loss/SRqqt3/assignment_loss‚1.0800924841985777 2loss/SRqqt4/assignment_loss‚1.8963399201631546 0loss/SRqqt4/detection_loss‚2.294087961741856 0loss/FBt2/assignment_loss‚0.1750656780121582 .loss/FRt2/detection_loss‚2.470711051708176 1loss/SRqqt3/detection_loss‚1.0754820404591057 .loss/FBt2/detection_loss‚2.650021391255515 0loss/FRt1/assignment_loss‚2.3542733629544577 1loss/SRqqt2/assignment_loss‚1.944341843594508 /loss/FBt1/detection_loss‚0.6461693874572392 trainer/global_step‚3799 1loss/FBt1/assignment_loss‚0.09564997408603244¸!ž2k’î‚(:‡ ž³éÔ¹•šm gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚172.118' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.0295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ð§í_ï‚(:Ô ž³éÔÁ¡ªn disk.md0.in‚ 1.94140625! disk.sdb2.out‚487.36865234375 disk.sda1.in‚0) proc.memory.percent‚18.38085651397705% disk./.usageGB‚2995.2589683532715 disk.sdb.in‚ 0.26953125 disk.sdb2.in‚ 0.26953125 disk.sda.in‚1.671875# proc.memory.rssMB‚ 3011.51953125 disk.md0.out‚ 468.515625 disk.rbd2.in‚0# disk.nvme0n1.out‚31006.76171875" disk.nvme1n1.out‚ 22770.1484375$ memory_percent‚95.97868919372559 cpu‚25.180998666754952 proc.cpu.threads‚17 disk.rbd0.in‚0 network.sent‚ 8.802341e+06 network.recv‚ 1.511843e+06 disk.dm-0.out‚53776.91015625 disk.sda2.in‚1.671875! disk.sda2.out‚487.36865234375 disk.rbd2.out‚1.125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0p1.out‚ 468.515625 disk.sdb.out‚487.36865234375 disk.sda1.out‚0' proc.memory.availableMB‚ 658.8515625 disk.dm-0.in‚ 72.6171875 disk.nvme1n1.in‚ 13.62109375 disk.sda.out‚487.36865234375 disk.rbd0.out‚1.125( disk./.usagePercent‚83.7486943684263 disk.md0p1.in‚ 1.94140625 disk.nvme0n1.in‚ 58.99609375ã{“ð‚(:ˆ ­³éÔɦìk gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚171.777' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.94425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Î@<#Zñ‚(:Ï ­³éÔ‘Òïl disk.md0p1.in‚ 1.94140625 disk.dm-0.in‚ 72.6171875# proc.memory.rssMB‚ 3011.51953125 cpu‚25.180465624431065 network.recv‚ 1.5162e+06 disk.sdb.in‚ 0.26953125 disk.md0.in‚ 1.94140625 disk.dm-0.out‚ 54006.359375 disk.nvme1n1.in‚ 13.62109375% disk./.usageGB‚2995.5781478881836 disk.md0p1.out‚ 470.109375 disk.sda2.in‚1.671875 disk.rbd0.in‚0 disk.sdb.out‚489.0341796875# disk.nvme1n1.out‚22770.26953125 disk.rbd2.in‚0 disk.sdb2.out‚489.0341796875 disk.sda1.out‚0 disk.rbd0.out‚1.125 disk.nvme0n1.in‚ 58.99609375# disk.nvme0n1.out‚31236.08984375 disk.sda.out‚489.0341796875( proc.memory.availableMB‚ 658.58984375 proc.cpu.threads‚17) disk./.usagePercent‚83.75761876180945 disk.sdb1.in‚0 disk.sda2.out‚489.0341796875 disk.md0.out‚ 470.109375 disk.sda.in‚1.671875 disk.sda1.in‚0$ memory_percent‚95.98028659820557 network.sent‚ 8.909157e+06 disk.rbd2.out‚1.125 disk.sdb2.in‚ 0.26953125) proc.memory.percent‚18.38085651397705 disk.sdb1.out‚0k‘G‘ò‚(:† ¼³éÔëê•m gpu.0.gpu‚54.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚123.712' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.928 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0À÷Gó‚(:Ò ¼³éÔô΄n disk.sda.in‚1.671875 disk.rbd0.out‚1.125 disk.sdb.in‚ 0.26953125 disk.nvme0n1.in‚ 58.99609375 proc.cpu.threads‚17 disk.md0.out‚ 472.17578125 disk.nvme1n1.in‚ 13.62109375) proc.memory.percent‚18.38085651397705 cpu‚25.179905493484288 disk.sda1.in‚0% disk./.usageGB‚2995.6422004699707 disk.sda.out‚491.18505859375 disk.sdb1.out‚0 disk.md0p1.out‚ 472.17578125! disk.nvme1n1.out‚ 22770.390625$ memory_percent‚95.98031044006348# proc.memory.rssMB‚ 3011.51953125 network.sent‚ 8.915186e+06 disk.sda1.out‚0 network.recv‚ 1.520968e+06 disk.rbd2.out‚1.125 disk.sdb2.in‚ 0.26953125 disk.md0p1.in‚ 1.94140625' proc.memory.availableMB‚ 658.5859375 disk.md0.in‚ 1.94140625 disk.rbd2.in‚0 disk.rbd0.in‚0! disk.sŒõÊ;da2.out‚491.18505859375 disk.dm-0.out‚ 54191.78125 disk.sda2.in‚1.671875! disk.nvme0n1.out‚ 31421.390625( disk./.usagePercent‚83.7594096988043! disk.sdb2.out‚491.18505859375 disk.dm-0.in‚ 72.6171875 disk.sdb1.in‚0 disk.sdb.out‚491.18505859375u¨2ô- _wandbruntime‚4478  _runtime‚4478–¥ÑÆõÀ $ _timestamp‚1.788500421169354e+09 _runtime‚4478.946570706 _step‚313 & lr-AdamW/pg1‚0.0009067601676377937 & lr-AdamW/pg2‚0.0009067601676377937 trainer/global_step‚3849´ñuËöÅ & lr-AdamW/pg2‚0.0009067601676377937 trainer/global_step‚3849 $ _timestamp‚1.788500421169354e+09 _runtime‚4478.946570706 _step‚313 & lr-AdamW/pg1‚0.0009067601676377937¹wfìâ2÷- _wandbruntime‚4478  _runtime‚44780Îrìñ'øëO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚362635786091008 trainer/global_step‚3849 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚328924556236288 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚328924556236288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚324512648649216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48264518 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚358024520229376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80551756 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97544944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4410997615616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23431818 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178070356 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97518600 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178096702 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97544944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚324512630646272 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80551758 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚362636047201792 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4394504711320 _step‚314 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚324512648649216 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚328923628261888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4411907587072 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71696334 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4611503032320 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 % _timestamp‚1.7885004211700888e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97518600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48264516 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _runtime‚ 4478.94742473 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚358024544169472 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚318164612201320 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80551758 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80551756 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚318165537215684 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚324512630646272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4611265861632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚313770124529360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71696962 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237170688 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚328923628261888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4410997615616 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚313770107490000 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4395412686324 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚324512630646272 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80551758 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚362636047201792 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4394504711320 _step‚314 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚324512648649216 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚328923628261888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4411907587072 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71696334 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4611503032320 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 % _timestamp‚1.7885004211700888e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97518600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48264516 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _runtime‚ 4478.94742473 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚358024544169472 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚318164612201320 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80551758 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80551756 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚318165537215684 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚324512630646272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4611265861632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚313770124529360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71696962 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237170688 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚328923628261888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4410997615616 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚313770107490000 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4395412686324 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚328924556236288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚324512648649216º”_-Ä2ú- _wandbruntime‚4480  _runtime‚4480! µ‚ü&ûöM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚313849125084544 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚329006368807936 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚324594249732096 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4413029060608 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚324594267735040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23437547 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚313849142123904 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80571916 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚329007296795648 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80571918 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329007296795648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4612658613248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4412119075840 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178141719 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚318245672083714 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4413029060608 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80571918 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71714760 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚362727167389696 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚358114745933824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97569801 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚324594267735040 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71714128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚324594249732096 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚358114769873920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178115347 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4412119075840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4612421455872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 trainer/global_step‚3849 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4395621984598 _step‚315 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97569801 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48276581 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 % _timestamp‚1.7885004222549322e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚318244747069142 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80571916 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚362727428487168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80571918 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329007296795648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4612658613248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4412119075840 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178141719 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚318245672083714 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 HBDeviceStatsMonitor.on_train_batch_end/overúxn×g!size_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4413029060608 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80571918 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71714760 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚362727167389696 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚358114745933824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97569801 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚324594267735040 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71714128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚324594249732096 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚358114769873920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178115347 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4412119075840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4612421455872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 trainer/global_step‚3849 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4395621984598 _step‚315 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97569801 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48276581 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 % _timestamp‚1.7885004222549322e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚318244747069142 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80571916 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚362727428487168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚324594249732096 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4413029060608 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚324594267735040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23437547 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚313849142123904 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80571916 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0»ožGÞ2ý- _wandbruntime‚4480  _runtime‚4480hÕ<áXþÒ 0loss/SRqqt4/assignment_loss‚2.24141125412037 1loss/SRqqt3/detection_loss‚0.9458406692961367 trainer/global_step‚3849 0loss/FRt4/assignment_loss‚3.8174391732071387 2loss/SRqqt2/assignment_loss‚1.7535805823281407 2loss/SRqqt3/assignment_loss‚1.1938264852567628 .loss/FRt2/detection_loss‚2.450890377956502 1loss/SRqqt1/detection_loss‚1.9609065130352974 .loss/FBt4/detection_loss‚2.487063476017543 &loss/total_loss‚44.636914488718105 $ _timestamp‚1.788500422257968e+09 /loss/FRt1/assignment_loss‚2.272311787347536 0loss/FRt3/assignment_loss‚1.9018521734646388 0loss/FBt2/assignment_loss‚0.2160922067748364 0loss/FBt3/assignment_loss‚0.1326519041839573 _step‚316 1loss/FBt1/assignment_loss‚0.14889010827792315 1loss/SRqqt2/detection_loss‚3.0664020660130875 2loss/SRqqt1/assignment_loss‚1.9797876723044152 0loss/FBt4/assignment_loss‚0.1407418958567957 /loss/FRt3/detection_loss‚1.8305159710229388 /loss/FRt4/detection_loss‚2.7249026840383355 /loss/FRt2/assignment_loss‚3.247952754193164 .loss/FBt3/detection_loss‚2.538786152998606 _runtime‚4480.035029711 epoch‚3 /loss/FRt1/detection_loss‚1.9059357056708854 1loss/SRqqt4/detection_loss‚2.4974885657429695 /loss/FBt1/detection_loss‚0.6855159193774064 .loss/FBt2/detection_loss‚2.496128390232722v±Z|]ÿ× 0loss/FRt4/assignment_loss‚3.8174391732071387 2loss/SRqqt2/assignment_loss‚1.7535805823281407 2loss/SRqqt3/assignment_loss‚1.1938264852567628 .loss/FRt2/detection_loss‚2.450890377956502 1loss/SRqqt1/detection_loss‚1.9609065130352974 .loss/FBt4/detection_loss‚2.487063476017543 &loss/total_loss‚44.636914488718105 $ _timestamp‚1.788500422257968e+09 /loss/FRt1/assignment_loss‚2.272311787347536 0loss/FRt3/assignment_loss‚1.9018521734646388 0loss/FBt2/assignment_loss‚0.2160922067748364 0loss/FBt3/assignment_loss‚0.1326519041839573 _step‚316 1loss/FBt1/assignment_loss‚0.14889010827792315 1loss/SRqqt2/detection_loss‚3.0664020660130875 2loss/SRqqt1/assignment_loss‚1.9797876723044152 0loss/FBt4/assignment_loss‚0.1407418958567957 /loss/FRt3/detection_loss‚1.8305159710229388 /loss/FRt4/detection_loss‚2.7249026840383355 /loss/FRt2/assignment_loss‚3.247952754193164 .loss/FBt3/detection_loss‚2.538786152998606 _runtime‚4480.035029711 epoch‚3 /loss/FRt1/detection_loss‚1.9059357056708854 1loss/SRqqt4/detection_loss‚2.4974885657429695 /loss/FBt1/detection_loss‚0.6855159193774064 .loss/FBt2/detection_loss‚2.496128390232722 0loss/SRqqt4/assignment_loss‚2.24141125412037 1loss/SRqqt3/detection_loss‚0.9458406692961367 trainer/global_step‚3849¼ôŒM‘€‚(:† ˳éÔâöþm gpu.0.gpu‚53.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚148.556' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.139 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0PB3rV‚(:Ë ˳éÔ†ò n disk.sdb.in‚ 0.26953125 disk.nvme0n1.in‚ 58.99609375 disk.rbd2.out‚1.125 disk.sdb2.in‚ 0.26953125 disk.md0p1.in‚ 1.94140625 disk.sda.in‚1.671875# proc.memory.rssMB‚ 3011.51953125$ disk./.usageGB‚2996.332546234131 disk.sdb2.out‚ 492.810546875 disk.sda.out‚ 492.810546875 disk.sda1.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sdb1.in‚0 network.recv‚ 1.52489e+06" disk.nvme0n1.out‚ 31473.4921875 disk.sda1.out‚0 disk.rbd0.out‚1.125 disk.sda2.in‚1.671875 disk.sda2.out‚ 492.810546875 disk.md0p1.out‚ 473.7265625 disk.dm-0.in‚ 72.6171875( proc.memory.availableMB‚ 658.78515625 disk.nvme1n1.in‚ 13.62109375 disk.md0.in‚ 1.94140625 disk.sdb1.out‚0 disk.dm-0.out‚ 54244.1015625$ memory_percent‚95.97909450531006 network.sent‚ 8.921073e+06) disk./.usagePercent‚83.77871205530238) proc.memory.percent‚18.38085651397705 disk.sdb.out‚ 492.810546875! disk.nvme1n1.out‚ 22770.609375 disk.md0.out‚ 473.7265625 disk.rbd2.in‚0 cpu‚25.179415436864257iœ“‚‚(:ˆ Ú³éԦŦm gpu.0.gpu‚40.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚164.649' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.16225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªñ¤Mƒ‚(: ڳéÔÅéÏm disk.sdb2.in‚ 0.26953125 disk.rbd0.out‚1.125 disk.sdb1.in‚0 proc.cpu.threads‚17 network.recv‚ 1.530834e+06 disk.md0.out‚ 475.26953125 disk.sda.out‚ 494.421875 disk.sdb1.out‚0# proc.memory.rssMB‚ 3011.51953125 disk.sda2.in‚1.671875 disk.rbd2.in‚0 disk.md0p1.out‚ 475.26953125 disk.nvme0n1.in‚ 58.99609375 disk.rbd0.in‚0( proc.memory.availableMB‚ 658.51171875 disk.sdb.in‚ 0.26953125 disk.md0.in‚ 1.94140625 disk.rbd2.out‚1.125) proc.memory.percent‚18.38085651397705 disk.sdb.out‚ 494.421875 disk.sda1.in‚0 cpu‚25.17888322704068 network.sent‚ 9.028996e+06( disk./.usagePercent‚83.7856576824936 disk.md0p1.in‚ 1.94140625 disk.dm-0.in‚ 72.6171875 disk.nvme1n1.in‚ 13.62109375 disk.sda.in‚1.671875$ memory_percent‚95.98076343536377 disk.sda2.out‚ 494.421875! disk.nvme0n1.out‚ 31759.890625 disk.sdb2.out‚ 494.421875 disk.sda1.out‚0$ disk./.usageGB‚2996.580955505371 disk.dm-0.out‚54530.66796875# disk.nvme1n1.out‚22770.77734375|ˆ™X„‚(:Í é³éÔÚ‰²o disk.dm-0.out‚ 54710.1171875 disk.sda2.out‚496.0419921875 disk.rbd2.out‚1.125 proc.cpu.threads‚17 network.sent‚ 9.034271e+06 disk.md0p1.in‚ 1.94140625# disk.nvme1n1.out‚22771.14453125 disk.sdb.out‚496.0419921875 disk.nvme0n1.in‚ 64.56640625 disk.sda.in‚1.671875 cpu‚25.178381981851903 disk.dm-0.in‚ 84.05078125 disk.md0.out‚ 476.81640625 disk.sdb2.out‚496.0419921875 disk.sda.out‚496.0419921875 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚1.125 disk.sdb1.in‚0 network.recv‚ 1.534341e+06 disk.sda2.in‚1.671875 disk.rbd2.in‚0 disk.rbd0.in‚0# memory_percent‚95.9808349609375# proc.memory.rssMB‚ 3011.51953125) proc.memory.percent‚18.38085651397705 disk.md0p1.out‚ 476.81640625# disk.nvme0n1.out‚31950.34765625 disk.nvme1n1.in‚ 19.484375 disk.md0.in‚ 1.94140625! proc.memory.availableMB‚658.5% disk./.usageGB‚2996.7050590515137 disk.sdb2.in‚ 0.26953125) disk./.usagePercent‚83.78912766958513 disk.sdb.in‚ 0.26953125 disk.sdb1.out‚0†ŒX…‚(:… é³éÔ¤©Ãp gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.992' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c¡œ“†‚(:ˆ ø³éÔöÕ’m gpu.0.gpu‚19.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚176.461' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.11525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UÐëM]‡‚(:Ò ø³éÔÀÍÇm disk.rbd0.out‚1.125# proc.memory.rssMB‚ 3011.51953125) proc.memory.percent‚18.38085651397705 disk.md0p1.out‚ 478.5234375 disk.sdb.out‚ 497.806640625 cpu‚25.177870560946026 disk.sda2.out‚ 497.806640625( proc.memory.availableMB‚ 658.58984375 disk.sda1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚83.78912713628247 disk.sdb.in‚ 0.26953125 disk.sda2.in‚1.671875 disk.nvme1n1.in‚ 77.8046875$ memory_percent‚95.98028659820557 network.sent‚ 9.040244e+06 disk.md0p1.in‚ 1.94140625 disk.sda.out‚ 497.806640625 proc.cpu.threads‚17% disk./.usageGB‚2996.7050399780273" disk.nvme1n1.out‚ 22887.6953125 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.dm-0.out‚55013.24609375 disk.nvme0n1.in‚ 102.37890625 disk.md0.out‚ 478.5234375 disk.rbd2.in‚0 disk.sdb2.in‚ 0.26953125 disk.sda.in‚1.671875 disk.rbd2.out‚1.125 network.recv‚ 1.538902e+06 disk.sdb1.in‚0# disk.nvme0n1.out‚32125.55078125 disk.md0.in‚ 1.94140625 disk.dm-0.in‚ 180.18359375 disk.sdb2.out‚ 497.806640625_SñÊ2ˆ- _wandbruntime‚4535  _runtime‚4535,þlƉÀ & lr-AdamW/pg1‚0.0009014754169325388 & lr-AdamW/pg2‚0.0009014754169325388 trainer/global_step‚3899 % _timestamp‚1.7885004773191946e+09 _runtime‚ 4535.09640946 _step‚317ñÆÒ;ËŠÅ & lr-AdamW/pg1‚0.0009014754169325388 & lr-AdamW/pg2‚0.0009014754169325388 trainer/global_step‚3899 % _timestamp‚1.7885004773191946e+09 _runtime‚ 4535.09640946 _step‚317½+$h€2‹- _wandbruntime‚4535  _runtime‚4535#IŠßò'ŒìO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚362535829391872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98789846 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚322171369183628 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4467995012096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81559758 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚367205702394368 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180323258 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚328593602940416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4467085040640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4451289941432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23721005 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98789846 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72589379 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48867768 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚333061597952512 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚606 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚362535805451776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚333060669978112 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48867766 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚367205942533632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚333061597952512 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚317721004256560 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚333060669978112 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98763502 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚322172294197992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚317720987217200 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4670113141760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81559756 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81559756 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4467995012096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180349604 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 % _timestamp‚1.7885004773203068e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚608 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4450381966428 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81559758 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚367205702394368 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180323258 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚328593602940416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4467085040640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4451289941432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23721005 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98789846 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72589379 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48867768 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚333061597952512 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚606 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚362535805451776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚333060669978112 @:DeviceStatsMonitor.on_train_batch_sDy¼êItart/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48867766 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚367205942533632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚333061597952512 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚317721004256560 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚333060669978112 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98763502 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚322172294197992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚317720987217200 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4670113141760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81559756 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81559756 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4467995012096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180349604 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 % _timestamp‚1.7885004773203068e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚608 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4450381966428 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3899 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚328675204023296 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180394621 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚328675222026240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23726743 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98814703 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚328675204023296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81579918 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚322251504051450 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚317800021851104 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 _step‚319 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81579916 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98814703 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48879831 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81579916 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚333143410524160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4469116485632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚333144338511872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81579918 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚328675222026240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚604 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72606574 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98788333 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23727347 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚367297110617088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚333144338511872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚4536.065142028 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180368249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚367297352840192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223104 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4468206500864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚362626055096320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48879833 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768àú¦¼'ûM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 _step‚319 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81579916 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98814703 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48879831 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81579916 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚333143410524160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4469116485632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚333144338511872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81579918 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚328675222026240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚604 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72606574 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98788333 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23727347 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚367297110617088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚333144338511872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚4536.065142028 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180368249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚367297352840192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223104 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4468206500864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚362626055096320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48879833 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚362626031156224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4452407214918 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4671297743872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98788333 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚606 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚317800004811744 % _timestamp‚1.7885004782873952e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3899 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚328675204023296 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180394621 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚328675222026240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23726743 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98814703 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚328675204023296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81579918 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚322251504051450 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚317800021851104¿_ÑQJ2‘- _wandbruntime‚4536  _runtime‚4536=Ÿ«V’Ð /loss/FRt3/assignment_loss‚1.493473901864021 1loss/SRqqt2/assignment_loss‚1.743751822755887 2loss/SRqqt4/assignment_loss‚1.5132511689745147 /loss/FBt1/detection_loss‚0.6546776223694906 /loss/FRt4/assignment_loss‚3.860975212910596 /loss/FRt1/detection_loss‚1.8614228720376713 %loss/total_loss‚45.14973490163028 $ _timestamp‚1.788500478289717e+09 0loss/FBt3/assignment_loss‚0.3214429276122246 1loss/SRqqt4/detection_loss‚2.4388977946906256 /loss/FBt3/detection_loss‚3.9943227767944336 trainer/global_step‚3899 1loss/SRqqt3/detection_loss‚1.1145921950846647 .loss/FBt2/detection_loss‚2.387575400727136 -loss/FBt4/detection_loss‚2.38515442609787 /loss/FRt3/detection_loss‚1.9530532495869743 1loss/SRqqt1/detection_loss‚1.9509250753455691 _runtime‚4536.066696946 .loss/FRt1/assignment_loss‚2.35017954288645 2loss/SRqqt1/assignment_loss‚1.8362686308445755 1loss/FBt2/assignment_loss‚0.30473344732729807 1loss/FBt4/assignment_loss‚0.45774442787944736 _step‚320 2loss/SRqqt3/assignment_loss‚1.1717678799697784 .loss/FRt2/detection_loss‚2.497905846425267 .loss/FRt4/detection_loss‚2.772231682258494 0loss/SRqqt2/detection_loss‚2.675926460669591 0loss/FRt2/assignment_loss‚3.3000577327816987 1loss/FBt1/assignment_loss‚0.10940280373600215 epoch‚3b§ì[“Õ 2loss/SRqqt1/assignment_loss‚1.8362686308445755 1loss/FBt2/assignment_loss‚0.30473344732729807 1loss/FBt4/assignment_loss‚0.45774442787944736 _step‚320 2loss/SRqqt3/assignment_loss‚1.1717678799697784 .loss/FRt2/detection_loss‚2.497905846425267 .loss/FRt4/detection_loss‚2.772231682258494 0loss/SRqqt2/detection_loss‚2.675926460669591 0loss/FRt2/assignment_loss‚3.3000577327816987 1loss/FBt1/assignment_loss‚0.10940280373600215 epoch‚3 /loss/FRt3/assignment_loss‚1.493473901864021 1loss/SRqqt2/assignment_loss‚1.743751822755887 2loss/SRqqt4/assignment_loss‚1.5132511689745147 /loss/FBt1/detection_loss‚0.6546776223694906 /loss/FRt4/assignment_loss‚3.860975212910596 /loss/FRt1/detection_loss‚1.8614228720376713 %loss/total_loss‚45.14973490163028 $ _timestamp‚1.788500478289717e+09 0loss/FBt3/assignment_loss‚0.3214429276122246 1loss/SRqqt4/detection_loss‚2.4388977946906256 /loss/FBt3/detection_loss‚3.9943227767944336 trainer/global_step‚3899 1loss/SRqqt3/detection_loss‚1.1145921950846647 .loss/FBt2/detection_loss‚2.387575400727136 -loss/FBt4/detection_loss‚2.38515442609787 /loss/FRt3/detection_loss‚1.9530532495869743 1loss/SRqqt1/detection_loss‚1.9509250753455691 _runtime‚4536.066696946 .loss/FRt1/assignment_loss‚2.35017954288645À“GìY‘”‚(:† ‡´éÔñËñl gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚177.688' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.422 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'Éûz^•‚(:Ó ‡´éÔꨙm proc.cpu.threads‚17% disk./.usageGB‚2996.0527153015137 disk.nvme0n1.in‚ 102.37890625 disk.sdb1.in‚0 disk.sdb1.out‚0$ memory_percent‚95.98050117492676) proc.memory.percent‚18.38085651397705 disk.sda1.out‚0 disk.nvme1n1.in‚ 77.8046875 disk.md0.in‚ 1.94140625 disk.sda2.in‚1.671875 disk.sdb2.out‚499.4443359375 disk.dm-0.out‚ 55429.875 cpu‚25.177400357233086 disk.sda2.out‚499.4443359375# disk.nvme1n1.out‚22887.93359375 disk.rbd0.in‚0' proc.memory.availableMB‚ 658.5546875 network.sent‚ 9.046066e+06 disk.sdb.in‚ 0.26953125# disk.nvme0n1.out‚32541.94140625 disk.sda1.in‚0 disk.rbd0.out‚1.125 network.recv‚ 1.543073e+06 disk.md0.out‚ 480.09765625 disk.rbd2.in‚0 disk.sda.in‚1.671875 disk.md0p1.in‚ 1.94140625 disk.md0p1.out‚ 480.09765625) disk./.usagePercent‚83.77088786530805 disk.sdb.out‚499.4443359375 disk.rbd2.out‚1.125# proc.memory.rssMB‚ 3011.51953125 disk.sdb2.in‚ 0.26953125 disk.sda.out‚499.4443359375 disk.dm-0.in‚ 180.18359375å• ’–‚(:‡ –´éÔ–ã¦m gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚181.082' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.2705 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áJdpQ—‚(:Æ –´éÔ°°ám disk.sdb.in‚ 0.26953125" disk.nvme0n1.out‚ 32713.5078125 disk.nvme1n1.in‚ 77.8046875 disk.sda.in‚1.671875 disk.rbd2.out‚1.125# memory_percent‚95.9782600402832& proc.memory.availableMB‚ 658.921875 cpu‚25.17692259184796 disk.md0p1.out‚ 481.7109375 disk.sdb1.out‚0 disk.sda2.in‚1.671875$ disk./.usageGB‚2996.052444458008 disk.sda1.out‚0 disk.dm-0.in‚ 180.18359375 proc.cpu.threads‚17 disk.sdb2.in‚ 0.26953125 disk.rbd0.in‚0 disk.sda2.out‚ 501.12109375 disk.nvme0n1.in‚ 102.37890625 disk.md0.out‚ 481.7109375 disk.dm-0.out‚ 55601.5546875! disk.nvme1n1.out‚ 22888.046875 disk.md0.in‚ 1.94140625 disk.rbd2.in‚0# proc.memory.rssMB‚ 3011.51953125 disk.sda1.in‚0 disk.rbd0.out‚1.125 disk.sdb1.in‚0 disk.md0p1.in‚ 1.94140625 disk.sdb.out‚ 501.12109375) proc.memory.percent‚18.38085651397705 network.sent‚ 9.155983e+06 network.recv‚ 1.548941e+06) disk./.usagePercent‚83.77088029241028 disk.sdb2.out‚ 501.12109375 disk.sda.out‚ 501.12109375Ž¯áÌ“˜‚(:ˆ ¥´éÔ·Žƒl gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.235' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.80875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤8L5噂(:Ö ¥´éÔÔé¶m disk.sdb.in‚ 0.26953125! disk.sda2.out‚502.88916015625 proc.cpu.threads‚17) disk./.usagePercent‚83.77164686165463 disk.sdb2.in‚ 0.26953125! disk.sdb2.out‚502.88916015625 disk.sda1.out‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 180.18359375# proc.memory.rssMB‚ 3011.51953125 network.recv‚ 1.552393e+06 disk.sda.out‚502.88916015625 disk.rbd0.in‚0 disk.rbd0.out‚1.125 disk.md0p1.out‚ 483.390625$ memory_percent‚95.98355293273926 cpu‚25.176458362509983 disk.sdb1.in‚0 disk.md0p1.in‚ 1.94140625 disk.dm-0.out‚55801.20703125 disk.sda2.in‚1.671875$ disk./.usageGB‚2996.079860687256 disk.md0.out‚ 483.390625 disk.nvme0n1.in‚ 102.3÷*|7890625" disk.nvme0n1.out‚ 32912.4765625' proc.memory.availableMB‚ 658.0546875 disk.md0.in‚ 1.94140625 disk.rbd2.in‚0 disk.rbd2.out‚1.125 network.sent‚ 9.159138e+06# disk.nvme1n1.out‚22888.73046875 disk.sdb.out‚502.88916015625) proc.memory.percent‚18.38085651397705 disk.nvme1n1.in‚ 77.8046875 disk.sda.in‚1.671875 disk.sda1.in‚0 wÝ‘š‚(:† ´´éÔëܬm gpu.0.gpu‚21.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚183.59' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.8975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qö[›‚(:Ð ´´éÔÒÏäm) disk./.usagePercent‚83.77258643428209 disk.rbd0.in‚0 disk.md0p1.in‚ 1.94140625 disk.md0.in‚ 1.94140625' proc.memory.availableMB‚ 658.4921875! proc.memory.rssMB‚ 3011.546875 network.recv‚ 1.556788e+06 disk.sda1.out‚0% disk./.usageGB‚2996.1134643554688 disk.sdb2.in‚ 0.26953125 disk.rbd0.out‚1.125 disk.dm-0.in‚ 180.18359375 disk.dm-0.out‚56071.33203125 disk.sdb.out‚504.7177734375 disk.md0.out‚ 485.15625 disk.rbd2.out‚1.125 network.sent‚ 9.167343e+06 disk.sdb2.out‚504.7177734375 disk.sda1.in‚0 disk.sdb1.in‚0# disk.nvme1n1.out‚22888.87109375 disk.sdb1.out‚0 disk.md0p1.out‚ 485.15625 disk.sda2.out‚504.7177734375$ memory_percent‚95.98088264465332 disk.sdb.in‚ 0.26953125 disk.sda2.in‚1.671875 disk.nvme0n1.in‚ 102.37890625 cpu‚25.175988967412206 disk.sda.in‚1.671875* proc.memory.percent‚18.381023406982422 proc.cpu.threads‚17 disk.sda.out‚504.7177734375" disk.nvme0n1.out‚ 33182.4609375 disk.nvme1n1.in‚ 77.8046875 disk.rbd2.in‚0¼ý„b2œ- _wandbruntime‚4590  _runtime‚4590,ºLÑÇÁ & lr-AdamW/pg1‚0.0008961124545721323 & lr-AdamW/pg2‚0.0008961124545721323 trainer/global_step‚3949 % _timestamp‚1.7885005329344974e+09 _runtime‚4590.711815502 _step‚321ÔX »ÌžÆ _runtime‚4590.711815502 _step‚321 & lr-AdamW/pg1‚0.0008961124545721323 & lr-AdamW/pg2‚0.0008961124545721323 trainer/global_step‚3949 % _timestamp‚1.7885005329344974e+09ÁÈŠ(2Ÿ- _wandbruntime‚4590  _runtime‚4590ÿö' ðO % _timestamp‚1.7885005329353492e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100034748 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393472 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4523172465664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚371775589857280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4507167196540 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚337197711694336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4506259221536 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚322 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚332674557231616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4728719576576 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚321671866944400 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82567756 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚332674539228672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4728499183104 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚326178126165936 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚371775834190848 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚337198639668736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49471018 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82567758 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚337197711694336 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4523172465664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82567758 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚604 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚337198639668736 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚332674539228672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73482775 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 182602506 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100008404 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100008404 _runtime‚4590.712687966 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚326179051180300 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182576160 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49471016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚367047114614272ÚÏãû'¡õO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚321671883983760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚367047090674176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4524082437120 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24012363 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24011759 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚332674557231616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚606 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚321671866944400 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82567756 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚332674539228672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4728499183104 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚326178126165936 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚371775834190848 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚337198639668736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49471018 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82567758 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚337197711694336 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4523172465664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82567758 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚604 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚337198639668736 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚332674539228672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73482775 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 182602506 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100008404 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100008404 _runtime‚4590.712687966 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚326179051180300 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182576160 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49471016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚367047114614272 % _timestamp‚1.7885005329353492e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100034748 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393472 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4523172465664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚371775589857280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4507167196540 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚337197711694336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4506259221536 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚322 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚332674557231616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4728719576576 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚611 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4729894505984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4508284470026 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 trainer/global_step‚3949 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚332756176317440 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82587916 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚367137316378624 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73500589 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 182621151 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4729665737216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚321750884538944 $ _timestamp‚1.788500533885081e+09 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82587916 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚323 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100059605 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100033235 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73501202 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚337280452240384 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100059605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚326258261033758 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚332756176317440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49483083 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚371867234824704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182647523 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚371866982115840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82587918 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49483081 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚332756158314496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚337281380228096 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4524293925888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82587918 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚613 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚367137340318720 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚337281380228096 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182621151 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚332756158314496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/oversizehÄz&Î_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚326259186048330 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚321750901578304˜‰“{'¤þM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24017508 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182647523 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚611 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4729894505984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4508284470026 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 trainer/global_step‚3949 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚332756176317440 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82587916 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚367137316378624 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73500589 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 182621151 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4729665737216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚321750884538944 $ _timestamp‚1.788500533885081e+09 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82587916 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚323 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100059605 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100033235 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73501202 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚337280452240384 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100059605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚326258261033758 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚332756176317440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49483083 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚371867234824704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182647523 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚371866982115840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82587918 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49483081 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚332756158314496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚337281380228096 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4524293925888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82587918 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚613 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚367137340318720 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚337281380228096 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182621151 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚332756158314496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚326259186048330 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚321750901578304çÖê,2¥- _wandbruntime‚4591  _runtime‚45918Œàƒ\¦Ö 1loss/SRqqt4/assignment_loss‚2.057543400261137 2loss/FBt3/assignment_loss‚0.007202863227576017 0loss/SRqqt1/detection_loss‚2.074615692550486 0loss/SRqqt2/detection_loss‚2.393589641366686 %loss/total_loss‚46.25490175807444 trainer/global_step‚3949 0loss/FBt2/assignment_loss‚0.2997609039768577 .loss/FBt3/detection_loss‚5.944643914699554 /loss/FBt4/detection_loss‚1.8994015753269196 epoch‚3 1loss/SRqqt2/assignment_loss‚1.544397568729307 0loss/FBt4/assignment_loss‚0.1711552671051777 /loss/FRt3/detection_loss‚2.0178147007165284 1loss/SRqqt4/detection_loss‚2.6394476338669106 % _timestamp‚1.7885005338874018e+09 0loss/FRt4/assignment_loss‚3.5208517221304088 1loss/FBt1/assignment_loss‚0.04684343210961395 /loss/FBt1/detection_loss‚0.8259144749536063 2loss/SRqqt3/assignment_loss‚1.2256546695180963 /loss/FRt1/detection_loss‚1.8689753751202192 /loss/FBt2/detection_loss‚2.5209211826324465 /loss/FRt4/detection_loss‚3.2627727618584266 0loss/SRqqt3/detection_loss‚1.002271856288467 _runtime‚4591.664395864 _step‚324 /loss/FRt2/detection_loss‚2.2782818801244913 0loss/FRt3/assignment_loss‚1.8653733000673096 2loss/SRqqt1/assignment_loss‚1.0348560851857518 0loss/FRt1/assignment_loss‚2.3520419964423547 0loss/FRt2/assignment_loss‚3.4005698598161036 ò.Óa§Û 0loss/SRqqt1/detection_loss‚2.074615692550486 0loss/SRqqt2/detection_loss‚2.393589641366686 %loss/total_loss‚46.25490175807444 trainer/global_step‚3949 0loss/FBt2/assignment_loss‚0.2997609039768577 .loss/FBt3/detection_loss‚5.944643914699554 /loss/FBt4/detection_loss‚1.8994015753269196 epoch‚3 1loss/SRqqt2/assignment_loss‚1.544397568729307 0loss/FBt4/assignment_loss‚0.1711552671051777 /loss/FRt3/detection_loss‚2.0178147007165284 1loss/SRqqt4/detection_loss‚2.6394476338669106 % _timestamp‚1.7885005338874018e+09 0loss/FRt4/assignment_loss‚3.5208517221304088 1loss/FBt1/assignment_loss‚0.04684343210961395 /loss/FBt1/detection_loss‚0.8259144749536063 2loss/SRqqt3/assignment_loss‚1.2256546695180963 /loss/FRt1/detection_loss‚1.8689753751202192 /loss/FBt2/detection_loss‚2.5209211826324465 /loss/FRt4/detection_loss‚3.2627727618584266 0loss/SRqqt3/detection_loss‚1.002271856288467 _runtime‚4591.664395864 _step‚324 /loss/FRt2/detection_loss‚2.2782818801244913 0loss/FRt3/assignment_loss‚1.8653733000673096 2loss/SRqqt1/assignment_loss‚1.0348560851857518 0loss/FRt1/assignment_loss‚2.3520419964423547 0loss/FRt2/assignment_loss‚3.4005698598161036 1loss/SRqqt4/assignment_loss‚2.057543400261137 2loss/FBt3/assignment_loss‚0.007202863227576017ÄÛarÅ“¨‚(:ˆ ôéÔÔÚÜm gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚180.473' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.11825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&b4Òb©‚(:× ôéÔÔ§–n network.recv‚ 1.560686e+06 disk.rbd2.in‚0 disk.sda.out‚506.6904296875( proc.memory.availableMB‚ 658.20703125 disk.nvme0n1.in‚ 102.37890625 disk.sdb1.out‚0 disk.md0p1.in‚ 1.94140625% disk./.usageGB‚2995.2783126831055 disk.nvme1n1.in‚ 77.8046875 disk.md0.in‚ 1.94140625 disk.rbd0.in‚0 disk.rbd0.out‚1.125 disk.md0.out‚ 487.05859375# disk.nvme0n1.out‚33350.60546875 disk.sdb2.out‚506.6904296875 disk.sda.in‚1.671875 disk.sda1.out‚0 network.sent‚ 9.170944e+06) disk./.usagePercent‚83.74923524398469 disk.sdb.in‚ 0.26953125 disk.sda2.in‚1.671875 disk.sdb1.in‚0! proc.memory.rssMB‚ 3011.546875* proc.memory.percent‚18.381023406982422 proc.cpu.threads‚17 disk.sdb.out‚506.6904296875" disk.nvme1n1.out‚ 22889.1796875 disk.sda1.in‚0 cpu‚25.175468422654085 disk.dm-0.in‚ 180.18359375 disk.dm-0.out‚56239.78515625 disk.sda2.out‚506.6904296875 disk.rbd2.out‚1.125 disk.sdb2.in‚ 0.26953125 disk.md0p1.out‚ 487.05859375$ memory_percent‚95.98262310028076ä"ÿ†“ª‚(:ˆ Ò´éÔ°´°m gpu.0.gpu‚54.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚175.235' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.80875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nBm»X«‚(:Í Ò´éÔ†ÕÎm disk.rbd2.out‚1.125 network.sent‚ 9.280759e+06 disk.sdb1.out‚0 disk.nvme0n1.in‚ 102.37890625 disk.sda.in‚1.671875 cpu‚25.17496212185691 network.recv‚ 1.565672e+06 disk.rbd0.in‚0 disk.sda2.out‚ 508.595703125 disk.nvme1n1.in‚ 77.8046875 disk.sda1.in‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 658.453125 disk.rbd2.in‚0 disk.sdb2.in‚ 0.26953125$ memory_percent‚95.98112106323242 disk.sdb1.in‚0 disk.sdb.in‚ 0.26953125# disk.nvme1n1.out‚22889.28515625$ disk./.usageGB‚2995.277820587158 disk.md0p1.in‚ 1.94140625 disk.md0p1.out‚ 488.8671875 disk.md0.out‚ 488.8671875! proc.memory.rssMB‚ 3011.546875 disk.dm-0.in‚ 180.18359375 disk.dm-0.out‚ 56369.0703125# disk.nvme0n1.out‚33479.78515625 disk.sdb2.out‚ 508.595703125 disk.sda.out‚ 508.595703125* proc.memory.percent‚18.381023406982422 proc.cpu.threads‚17) disk./.usagePercent‚83.74922148477604 disk.rbd0.out‚1.125 disk.sdb.out‚ 508.595703125 disk.sda2.in‚1.671875 disk.md0.in‚ 1.94140625ÕOhl’¬‚(:‡ á´éԦŎm gpu.0.gpu‚22.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚181.414' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.3535 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓÚÛ¦O­‚(:Ä á´éÔþ¹¹m disk.sdb2.in‚ 0.26953125 disk.sdb2.out‚ 510.3203125 disk.sda.in‚1.671875! proc.memory.rssMB‚ 3011.546875 network.sent‚ 9.283168e+06 disk.rbd0.out‚1.125 disk.dm-0.in‚ 180.18359375 disk.sdb.out‚ 510.3203125 cpu‚25.174400812282574 disk.sdb.in‚ 0.26953125 disk.md0.in‚ 1.94140625 disk.rbd2.out‚1.125* proc.memory.percent‚18.381023406982422% disk./.usageGB‚2995.4372596740723 disk.rbd0.in‚0 disk.sda2.in‚1.671875( proc.memory.availableMB‚ 658.51171875 disk.sdb1.in‚0 disk.md0p1.out‚ 490.53515625" disk.nvme0n1.out‚ 33532.6796875 disk.rbd2.in‚0$ memory_percent‚95.98076343536377 disk.sda1.in‚0 disk.md0p1.in‚ 1.94140625 disk.dm-0.out‚ 56422.2734375 disk.sda2.out‚ 510.3203125 disk.nvme1n1.in‚ 77.8046875 disk.md0.out‚ 490.53515625 disk.sda.out‚ 510.3203125 proc.cpu.threads‚17( disk./.usagePercent‚83.7536794683766 disk.sda1.out‚0 disk.sdb1.out‚0 disk.nvme1n1.out‚ 22889.59375 network.recv‚ 1.5688e+06 disk.nvme0n1.in‚ 102.37890625º3u×2®- _wandbruntime‚4642  _runtime‚4642'•¼ÀǯÁ & lr-AdamW/pg1‚0.0008906723981898922 & lr-AdamW/pg2‚0.0008906723981898922 trainer/global_step‚3999 % _timestamp‚1.7885005847129803e+09 _runtime‚4642.490516901 _step‚325Dý*PÌ°Æ % _timestamp‚1.7885005847129803e+09 _runtime‚4642.490516901 _step‚325 & lr-AdamW/pg1‚0.0008906723981898922 & lr-AdamW/pg2‚0.0008906723981898922 trainer/global_step‚3999Å"CGô2±- _wandbruntime‚4642  _runtime‚4642ºÌÝö'²ðO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _step‚326 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83575756 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 % _timestamp‚1.7885005847145524e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101279650 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚341334753410560 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚341335681384960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚325622746671600 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4562136476644 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚341334753410560 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4579259890688 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74376995 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4787531259392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚336755493519872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50074268 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 184855408 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚325622763710960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184829062 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4579259890688 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚371558375896576 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101253306 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4563044451648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101279650 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83575758 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101253306 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24302727 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚341335681384960 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4580169862144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24302094 trainer/global_step‚3999 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚330184883148244 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚4642.491929868 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚336755493519872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4580169862144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0]P"À³õO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚376345931096064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚336755511522816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50074266 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83575758 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4580169862144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24302094 trainer/global_step‚3999 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚330184883148244 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚4642.491929868 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚336755493519872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4580169862144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ZLDeviceStatsMonitor.on_traiq­ ½án_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _step‚326 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83575756 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 % _timestamp‚1.7885005847145524e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101279650 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚341334753410560 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚341335681384960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚325622746671600 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4562136476644 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚341334753410560 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4579259890688 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74376995 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4787531259392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚336755493519872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50074268 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 184855408 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚325622763710960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184829062 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4579259890688 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚371558375896576 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101253306 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4563044451648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101279650 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83575758 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101253306 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24302727 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚341335681384960 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚336837130608640 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83595916 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101304507 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50086331 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24307927 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184900425 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4580381350912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101278137 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74394901 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4788694557184 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚325701764266144 % _timestamp‚1.7885005856919756e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184874053 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚327 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚336837112605696 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50086333 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24308568 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚336837130608640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74394258 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184900425 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 184874053 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚371648601601024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4564161725134 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚341418421944320 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚376437320098304 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚325701781305504 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚376437063195136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4563253749922 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4580381350912 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚371648625541120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101304507 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570Q}|'¶ÿM EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83595916 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101304507 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50086331 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24307927 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184900425 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4580381350912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101278137 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74394901 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4788694557184 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚325701764266144 % _timestamp‚1.7885005856919756e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184874053 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚327 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚336837112605696 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50086333 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24308568 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚336837130608640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74394258 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184900425 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 184874053 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚371648601601024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4564161725134 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚341418421944320 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚376437320098304 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚325701781305504 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚376437063195136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4563253749922 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4580381350912 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚371648625541120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101304507 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4788461594112 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚330265943030638 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83595918 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4581291335680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83595918 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101278137 _runtime‚4643.470078793 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚341417493956608 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚336837130608640 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0Ǧs2·- _wandbruntime‚4643  _runtime‚4643Ù™Ü3X¸Ò /loss/FRt1/detection_loss‚1.6297921556839288 .loss/FRt2/detection_loss‚2.676746506872587 0loss/FRt3/assignment_loss‚1.4166810828627963 0loss/FRt4/assignment_loss‚2.6449282920878865 0loss/FBt1/assignment_loss‚0.1172192829621648 /loss/FBt1/detection_loss‚0.6302359158063738 0loss/FBt3/assignment_loss‚0.0972023283959263 0loss/FRt2/assignment_loss‚3.5290490657091143 2loss/SRqqt1/assignment_loss‚1.7882141347687979 2loss/SRqqt2/assignment_loss‚1.9400186554893204 .loss/FBt4/detection_loss‚3.571774979432424 0loss/FRt1/assignment_loss‚1.9616574909951952 1loss/SRqqt2/detection_loss‚2.5593387821446294 .loss/FBt2/detection_loss‚2.837854521142112 _runtime‚4643.471045668 2loss/SRqqt4/assignment_loss‚1.7759941925897318 /loss/FBt4/assignment_loss‚0.552129776254939 0loss/SRqqt1/detection_loss‚1.972734884573863 /loss/FBt3/detection_loss‚2.1873422662417092 trainer/global_step‚3999 _step‚328 1loss/SRqqt3/assignment_loss‚1.069715730752796 epoch‚3 % _timestamp‚1.7885005856940176e+09 1loss/FBt2/assignment_loss‚0.22527313166129817 /loss/FRt3/detection_loss‚1.7295412482185797 /loss/FRt4/detection_loss‚3.4076720579810766 0loss/SRqqt3/detection_loss‚1.085282551378689 0loss/SRqqt4/detection_loss‚2.542288487448412 %loss/total_loss‚43.94868752145435ðÑ^ä]¹× 0loss/FRt4/assignment_loss‚2.6449282920878865 0loss/FBt1/assignment_loss‚0.1172192829621648 /loss/FBt1/detection_loss‚0.6302359158063738 0loss/FBt3/assignment_loss‚0.0972023283959263 0loss/FRt2/assignment_loss‚3.5290490657091143 2loss/SRqqt1/assignment_loss‚1.7882141347687979 2loss/SRqqt2/assignment_loss‚1.9400186554893204 .loss/FBt4/detection_loss‚3.571774979432424 0loss/FRt1/assignment_loss‚1.9616574909951952 1loss/SRqqt2/detection_loss‚2.5593387821446294 .loss/FBt2/detection_loss‚2.837854521142112 _runtime‚4643.471045668 2loss/SRqqt4/assignment_loss‚1.7759941925897318 /loss/FBt4/assignment_loss‚0.552129776254939 0loss/SRqqt1/detection_loss‚1.972734884573863 /loss/FBt3/detection_loss‚2.1873422662417092 trainer/global_step‚3999 _step‚328 1loss/SRqqt3/assignment_loss‚1.069715730752796 epoch‚3 % _timestamp‚1.7885005856940176e+09 1loss/FBt2/assignment_loss‚0.22527313166129817 /loss/FRt3/detection_loss‚1.7295412482185797 /loss/FRt4/detection_loss‚3.4076720579810766 0loss/SRqqt3/detection_loss‚1.085282551378689 0loss/SRqqt4/detection_loss‚2.542288487448412 %loss/total_loss‚43.94868752145435 /loss/FRt1/detection_loss‚1.6297921556839288 .loss/FRt2/detection_loss‚2.676746506872587 0loss/FRt3/assignment_loss‚1.4166810828627963Èu§3ú’º‚(:‡ ð´éÔæ ôl gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚158.746' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.6865 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð.9T»‚(:É ð´éÔ§¢Êm disk.nvme0n1.in‚ 103.21484375 disk.rbd2.out‚1.125 disk.sda1.out‚0 disk.dm-0.out‚56649.36328125* proc.memory.percent‚18.381023406982422 network.sent‚ 9.289084e+06 disk.sdb2.out‚ 511.833984375$ memory_percent‚95.97947597503662! proc.memory.rssMB‚ 3011.546875 disk.nvme1n1.in‚ 77.8046875( proc.memory.availableMB‚ 658.72265625 disk.rbd0.in‚0( disk./.usagePercent‚83.7576175885436 disk.md0.in‚ 1.94140625 disk.md0.out‚ 491.984375 disk.sda.in‚1.671875 disk.sda1.in‚0 network.recv‚ 1.573458e+06 disk.sdb2.in‚ 0.26953125 disk.sdb1.in‚0 cpu‚25.17389322212028 disk.sdb.in‚ 0.26953125# disk.nvme1n1.out‚22889.73828125 disk.rbd2.in‚0 disk.md0p1.in‚ 1.94140625 disk.md0p1.out‚ 491.984375 disk.dm-0.in‚ 181.01953125 disk.sdb.out‚ 511.833984375 disk.nvme0n1.out‚ 33759.625 disk.sdb1.out‚0 disk.rbd0.out‚1.125% disk./.usageGB‚2995.5781059265137 disk.sda2.out‚ 511.833984375 disk.sda.out‚ 511.833984375 proc.cpu.threads‚17 disk.sda2.in‚1.671875¿d“¼‚(:ˆ ÿ´éÔ¯¿”m gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚178.117' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.52925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~»)Ød½‚(:Ù ÿ´éÔõ³Æm disk.sda2.in‚1.671875 disk.rbd2.in‚0! disk.sdb2.out‚513.78662109375 disk.sdb1.in‚0 disk.dm-0.in‚ 181.01953125 disk.dm-0.out‚56758.03515625* proc.memory.percent‚18.381023406982422 network.sent‚ 9.396646e+06 disk.nvme1n1.in‚ 77.8046875 disk.sdb.out‚513.78662109375! disk.sda2.out‚513.78662109375' proc.memory.availableMB‚ 658.7265625 disk.sda.in‚1.671875 disk.md0p1.in‚ 1.94140625 disk.md0.in‚ 1.94140625$ memory_percent‚95.97945213317871 cpu‚25.17344398975192 disk.sdb2.in‚ 0.26953125# disk.nvme1n1.out‚22890.04296875 disk.rbd0.out‚1.125 disk.md0p1.out‚ 493.86328125 disk.sdb.in‚ 0.26953125 disk.md0.out‚ 493.86328125! proc.memory.rssMB‚ 3011.546875 proc.cpu.threads‚17) disk./.usagePercent‚83.74298525678395 disk.sda1.out‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 103.21484375" disk.nvme0n1.out‚ 33867.9921875 disk.rbd2.out‚1.125 network.recv‚ 1.578713e+06% disk./.usageGB‚2995.0547828674316 disk.sda.out‚513.78662109375 disk.sda1.in‚0 disk.sdb1.out‚0U¸-Ò‘¾‚(:† ŽµéÔ»µ¸m gpu.0.gpu‚57.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚141.704' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.426 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U²b8¿‚(:Ö ŽµéÔäµ¥n disk.rbd0.out‚ 1.13671875 disk.sdb1.out‚0" disk.nvme1n1.out‚ 22890.1640625 disk.sda.out‚515.56689453125$ disk./.usageGB‚2995.116916656494 disk.sdb.out‚515.56689453125 disk.md0.out‚495.5625 proc.cpu.threads‚17 network.sent‚ 9.403e+06 network.recv‚ 1.583248e+06 disk.sda1.out‚0! disk.nvme0n1.out‚ 33968.515625 disk.rbd2.out‚ 1.13671875 disk.rbd2.in‚0 disk.sdb2.in‚ 0.26953125 disk.rbd0.in‚0! disk.sda2.out‚515.56689453125 disk.nvme0n1.in‚ 103.21484375 disk.md0.in‚BA¸ƒ) 1.94140625 disk.sda.in‚1.671875( proc.memory.availableMB‚ 658.47265625 disk.sda1.in‚0 disk.sda2.in‚1.671875! disk.sdb2.out‚515.56689453125 disk.md0p1.in‚ 1.94140625 disk.sdb1.in‚0 disk.dm-0.in‚ 181.01953125 disk.sdb.in‚ 0.26953125$ memory_percent‚95.98100185394287! proc.memory.rssMB‚ 3011.546875* proc.memory.percent‚18.381023406982422 cpu‚25.172935972976788) disk./.usagePercent‚83.74472254353115 disk.md0p1.out‚495.5625 disk.dm-0.out‚ 56858.6796875 disk.nvme1n1.in‚ 77.8046875^C5À‚(:’ µéԌەm gpu.0.gpu‚49.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚119.368' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.841999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-2éïdÁ‚(:Ù µéÔ‰ƒðm disk.sda2.in‚1.671875 network.sent‚ 9.408294e+06$ disk./.usageGB‚2995.329936981201! disk.sdb2.out‚517.00634765625 disk.sda1.out‚0 disk.sdb1.out‚0 disk.md0p1.in‚ 1.94140625 disk.md0p1.out‚ 496.9453125# disk.nvme1n1.out‚22890.63671875 disk.sda.out‚517.00634765625 disk.dm-0.in‚ 181.01953125 disk.nvme0n1.in‚ 103.21484375# disk.nvme0n1.out‚34109.36328125* proc.memory.percent‚18.381023406982422 cpu‚25.172496961786084 network.recv‚ 1.587199e+06 disk.nvme1n1.in‚ 77.8046875 disk.md0.in‚ 1.94140625 disk.rbd2.out‚ 1.13671875 disk.sda1.in‚0 disk.rbd0.in‚0! proc.memory.rssMB‚ 3011.546875 disk.sdb.in‚ 0.26953125 disk.md0.out‚ 496.9453125 disk.sda.in‚1.671875 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd0.out‚ 1.19921875 disk.sdb1.in‚0 disk.dm-0.out‚57000' proc.memory.availableMB‚ 658.6953125 disk.sdb2.in‚ 0.26953125 disk.sdb.out‚517.00634765625! disk.sda2.out‚517.00634765625$ memory_percent‚95.97964286804199) disk./.usagePercent‚83.75067868096589Oõ¼‡2Â- _wandbruntime‚4700  _runtime‚4700ÄZ ~ÇÃÁ & lr-AdamW/pg2‚0.0008851563814854143 trainer/global_step‚4049 % _timestamp‚1.7885006423835685e+09 _runtime‚4700.160817125 _step‚329 & lr-AdamW/pg1‚0.0008851563814854143»LØÝÌÄÆ _runtime‚4700.160817125 _step‚329 & lr-AdamW/pg1‚0.0008851563814854143 & lr-AdamW/pg2‚0.0008851563814854143 trainer/global_step‚4049 % _timestamp‚1.7885006423835685e+09É´4Ö2Å- _wandbruntime‚4700  _runtime‚4700Êuõ'ÆïO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4845786196480 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚380915447315456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4635347315712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚635 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4618921706756 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚340836447811072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚340836465814016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚340836447811072 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _step‚330 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102498208 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 187108310 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 187081964 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237170688 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84583756 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚345471795126784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24592966 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚345471795126784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4636257287168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24593599 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚334192565144916 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102498208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75271117 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚340836465814016 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚329573626398800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 % _timestamp‚1.7885006423845959e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84583758 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84583756 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376069685059072 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4618013731752 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚334191640130552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚376069661118976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84583758 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚345472723101184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50677518 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚345472723101184 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4846023367168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚329573643438160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102524552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75270482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102524552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 trainer/global_step‚4049σy~ú'ÇôO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚340836447811072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚340836465814016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚340836447811072 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _step‚330 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102498208 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 187108310 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 187081964 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237170688 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84583756 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚345471795126784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24592966 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚345471795126784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4636257287168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24593599 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚334192565144916 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102498208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75271117 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚340836465814016 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚329573626398800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 % _timestamp‚1.7885006423845959e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84583758 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84583756 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376069685059072 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4618013731752 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚334191640130552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚376069661118976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84583758 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚345472723101184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50677518 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚345472723101184 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4846023367168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚329573643438160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102524552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75270482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102524552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 trainer/global_step‚4049 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4845786196480 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚380915447315456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4635347315712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚635 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4618921706756 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344ÊæËúñ2È- _wandbruntime‚4701  _runtime‚4701¸kq%ÉùM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50689581 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚376159910763520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4636468775936 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75288395 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚331 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102523039 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187153327 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187153327 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84603918 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚345555463660544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75289018 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 $ _timestamp‚1.788500643532938e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24599435 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚329652643993344 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚329652661032704 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84603916 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚340918066896896 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚381007099110912 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102549409 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 187126955 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4636468775936 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102523039 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4637378760704 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚345555463660544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚334272700012946 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4620038980242 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 trainer/global_step‚4049 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102549409 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚340918084899840 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚340918066896896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4846963772928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚334271774998374 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚345554535672832 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84603916 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _runtime‚4701.310884749 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚381006850596352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 187126955 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84603918 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50689583 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚381006850596352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 187126955 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84603918 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50689583 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75288395 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚331 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102523039 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187153327 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187153327 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84603918 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚345555463660544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75289018 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 $ _timestamp‚1.788500643532938e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24599435 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚329652643993344 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚329652661032704 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84603916 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚340918066896896 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚381007099110912 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102549409 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 187126955 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4636468775936 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102523039 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4637378760704 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚345555463660544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚334272700012946 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4620038980242 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 trainer/global_step‚4049 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102549409Ë’¼c»2Ë- _wandbruntime‚4701  _runtime‚4701²ŽR€ZÌÔ 0loss/FRt1/assignment_loss‚2.5111931959788003 0loss/FBt3/assignment_loss‚0.4208734152489342 /loss/FBt2/detection_loss‚2.5611504444054196 /loss/FBt3/detection_loss‚2.6611017286777496 _runtime‚4701.311881114 2loss/SRqqt2/assignment_loss‚1.4566967299975015 1loss/SRqqt3/detection_loss‚0.9047532148476018 _step‚332 0loss/FRt3/assignment_loss‚1.9090080959456308 1loss/SRqqt3/assignment_loss‚1.065319853865343 /loss/FBt1/assignment_loss‚0.165987313589517 /loss/FRt4/detection_loss‚3.0179256836573285 /loss/FBt4/detection_loss‚2.1560740967591605 /loss/FRt2/assignment_loss‚3.257197094518085 0loss/FBt2/assignment_loss‚0.2047091326156598 /loss/FRt1/detection_loss‚1.9185241303008904 1loss/SRqqt1/detection_loss‚1.6829372508959337 0loss/SRqqt2/detection_loss‚2.456419644110343 &loss/total_loss‚43.057990184257676 1loss/SRqqt4/detection_loss‚2.6198939866033095 0loss/FRt4/assignment_loss‚3.5971882303555804 2loss/SRqqt4/assignment_loss‚1.7532090277507388 /loss/FBt1/detection_loss‚0.9289793586084509 epoch‚3 $ _timestamp‚1.788500643534843e+09 .loss/FRt2/detection_loss‚2.327923875908519 2loss/SRqqt1/assignment_loss‚1.2689518086950888 0loss/FBt4/assignment_loss‚0.5457710323904242 /loss/FRt3/detection_loss‚1.6662018385316644 trainer/global_step‚4049V’8_ÍÙ 2loss/SRqqt2/assignment_loss‚1.4566967299975015 1loss/SRqqt3/detection_loss‚0.9047532148476018 _step‚332 0loss/FRt3/assignment_loss‚1.9090080959456308 1loss/SRqqt3/assignment_loss‚1.065319853865343 /loss/FBt1/assignment_loss‚0.165987313589517 /loss/FRt4/detection_loss‚3.0179256836573285 /loss/FBt4/detection_loss‚2.1560740967591605 /loss/FRt2/assignment_loss‚3.257197094518085 0loss/FBt2/assignment_loss‚0.2047091326156598 /loss/FRt1/detection_loss‚1.9185241303008904 1loss/SRqqt1/detection_loss‚1.6829372508959337 0loss/SRqqt2/detection_loss‚2.456419644110343 &loss/total_loss‚43.057990184257676 1loss/SRqqt4/detection_loss‚2.6198939866033095 0loss/FRt4/assignment_loss‚3.5971882303555804 2loss/SRqqt4/assignment_loss‚1.7532090277507388 /loss/FBt1/detection_loss‚0.9289793586084509 epoch‚3 $ _timestamp‚1.788500643534843e+09 .loss/FRt2/detection_loss‚2.327923875908519 2loss/SRqqt1/assignment_loss‚1.2689518086950888 0loss/FBt4/assignment_loss‚0.5457710323904242 /loss/FRt3/detection_loss‚1.6662018385316644 trainer/global_step‚4049 0loss/FRt1/assignment_loss‚2.5111931959788003 0loss/FBt3/assignment_loss‚0.4208734152489342 /loss/FBt2/detection_loss‚2.5611504444054196 /loss/FBt3/detection_loss‚2.6611017286777496 _runtime‚4701.311881114Ì¢öò¥g΂(:Ü ¬µéÔ«‡øl# disk.nvme0n1.out‚34253.20703125 network.sent‚ 9.414282e+06 disk.md0.in‚ 1.94140625 disk.rbd0.out‚ 1.19921875 disk.sdb1.in‚0 disk.dm-0.out‚57143.98828125 disk.sda2.in‚1.671875! disk.sda2.out‚519.16748046875! disk.sdb2.out‚519.16748046875 disk.nvme1n1.in‚ 77.8046875* proc.memory.percent‚18.381023406982422 proc.cpu.threads‚17 network.recv‚ 1.591923e+06 disk.sda1.out‚0 disk.rbd0.in‚0! proc.memory.rssMB‚ 3011.546875 disk.sda.in‚1.671875 disk.sda.out‚519.16748046875 disk.nvme1n1.out‚ 22890.78125 cpu‚25.172066572162382 disk.nvme0n1.in‚ 104.1796875) disk./.usagePercent‚83.75860387848412 disk.md0.out‚ 499.0390625 disk.rbd2.in‚0 disk.sdb2.in‚ 0.26953125 disk.md0p1.in‚ 1.94140625 disk.md0p1.out‚ 499.0390625 disk.sdb.out‚519.16748046875$ disk./.usageGB‚2995.613380432129 disk.rbd2.out‚ 1.19921875 disk.sda1.in‚0 disk.sdb1.out‚0 disk.sdb.in‚ 0.26953125 disk.dm-0.in‚ 181.984375$ memory_percent‚95.98097801208496' proc.memory.availableMB‚ 658.4765625z¬{“Ï‚(:ˆ ¬µéÔÁå¯m gpu.0.gpu‚52.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.157' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.78925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yÁ$“Ђ(:ˆ »µéÔ⯅m gpu.0.gpu‚41.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.891' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.72275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kž˜hÑ‚(:Ý »µéÔ§„¡n network.recv‚ 1.597054e+06 disk.sdb1.out‚0 disk.sda.in‚1.671875 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚18.381023406982422 proc.cpu.threads‚17 network.sent‚ 9.523778e+06% disk./.usageGB‚2996.3345222473145 disk.nvme0n1.in‚ 104.1796875 disk.md0p1.out‚ 500.53515625 disk.nvme1n1.in‚ 77.8046875( proc.memory.availableMB‚ 658.71484375 disk.md0p1.in‚ 1.94140625 disk.dm-0.in‚ 181.984375 disk.dm-0.out‚ 57432.1328125! disk.nvme1n1.out‚ 22891.296875 disk.sdb2.out‚520.7373046875 disk.sdb.out‚520.7373046875 disk.sda2.in‚1.671875 disk.rbd2.in‚0 disk.sdb1.in‚0" disk.nvme0n1.out‚ 34540.8359375 disk.md0.in‚ 1.94140625 disk.md0.out‚ 500.53515625 disk.rbd2.out‚ 1.19921875 disk.sdb2.in‚ 0.26953125 disk.sda.out‚520.7373046875 disk.rbd0.out‚ 1.2783203125) disk./.usagePercent‚83.77876730545802 disk.sdb.in‚ 0.26953125! proc.memory.rssMB‚ 3011.546875 cpu‚25.171562078556843 disk.sda2.out‚520.7373046875 disk.rbd0.in‚0$ memory_percent‚95.97952365875244Jch“Ò‚(:ˆ ʵéÔ²Ž€m gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.383' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.09575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09ؼvjÓ‚(:ß ʵéÔ†ôãm disk.rbd2.out‚ 1.2783203125 disk.sdb2.in‚ 0.26953125! disk.sdb2.out‚522.30712890625 disk.md0p1.in‚ 1.94140625 disk.sda2.in‚1.671875# memory_percent‚95.9794282913208 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 181.984375! disk.sda2.out‚522.30712890625# disk.nvme0n1.out‚34774.81640625 disk.nvme1n1.in‚ 77.8046875 disk.sda.in‚1.671875 disk.rbd0.out‚ 1.2783203125 network.sent‚ 9.530098e+06 disk.rbd2.in‚0 disk.sda1.in‚0 cpu‚25.171131398998934 network.recv‚ 1.601635e+06# disk./.usageGB‚2996.33198928833# disk.nvme1n1.out‚22891.40234375 disk.md0p1.out‚ 502.0390625 disk.dm-0.out‚ 57666.21875( proc.memory.availableMB‚ 658.73046875 disk.sda.out‚522.30712890625 disk.sda1.out‚0 disk.sdb.in‚ 0.26953125! proc.memory.rssMB‚ 3011.546875 proc.cpu.threads‚17) disk./.usagePercent‚83.77869648286469 disk.nvme0n1.in‚ 104.1796875 disk.md0.in‚ 1.94140625 disk.md0.out‚ 502.0390625 disk.sdb1.in‚0 disk.sdb.out‚522.30712890625* proc.memory.percent‚18.381023406982422åXTL‘Ô‚(:† ÙµéÔÔ²§m gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚177.508' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.377 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï¯{ÛfÕ‚(:Û ÙµéÔ߯ôm disk.nvme0n1.in‚ 104.1796875( proc.memory.availableMB‚ 658.70703125 disk.md0.in‚ 1.94140625 disk.rbd2.in‚0 disk.sda.in‚1.671875* proc.memory.percent‚18.381023406982422! disk.sdb2.out‚523.60498046875 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.md0p1.in‚ 1.94140625 disk.sda1.out‚0 disk.dm-0.out‚ 57912.09375 disk.sdb.out‚523.60498046875 disk.sda2.in‚1.671875 network.sent‚ 9.533152e+06# disk.nvme1n1.out‚22891.51171875 disk.md0.out‚ 503.28125 disk.sda1.in‚0 disk.rbd0.out‚ 1.34130859375 disk.sdb.in‚ 0.26953125! disk.sda2.out‚523.60498046875 disk.sda.out‚523.60498046875 network.recv‚ 1.605057e+06) disk./.usagePercent‚83.78571506585989 disk.sdb1.out‚0# disk.nvme0n1.out‚35020.58203125 disk.rbd2.out‚ 1.2783203125$ memory_percent‚95.97957134246826! proc.memory.rssMB‚ 3011.546875 disk.md0p1.out‚ 503.28125 disk.dm-0.in‚ 181.984375 disk.nvme1n1.in‚ 77.8046875 disk.sdb2.in‚ 0.26953125 cpu‚25.17064591328677 proc.cpu.threads‚17" disk./.usageGB‚2996.5830078125íß„À2Ö- _wandbruntime‚4757  _runtime‚4757…·î@Æ×À & lr-AdamW/pg1‚0.0008795655539883124 & lr-AdamW/pg2‚0.0008795655539883124 trainer/global_step‚4099 % _timestamp‚1.7885006998968506e+09 _runtime‚ 4757.67425699 _step‚333ÌxåbËØÅ & lr-AdamW/pg1‚0.0008795655539883124 & lr-AdamW/pg2‚0.0008795655539883124 trainer/global_step‚4099 % _timestamp‚1.7885006998968506e+09 _runtime‚ 4757.67425699 _step‚333Ís {2Ù- _wandbruntime‚4757  _runtime‚4757Õ|ö'ÚðO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189361212 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103769454 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚344917420105216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4904528924672 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚338199322127224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85591756 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4691434740736 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚349608836843008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85591756 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚385485475266048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚344917402102272 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚385485736376832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚333524523165360 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51280768 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349608836843008 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚344917420105216 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4692344712192 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103743110 trainer/global_step‚4099 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚4757.675751437 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚349609764817408 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4692344712192 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚333524523165360 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51280768 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349608836843008 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚344917420105216 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4692344712192 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103743110 trainer/global_step‚4099 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚4757.675751437 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚349609764817408 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4692344712192 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚349608836843008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85591756 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚385485475266048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚344917402102272 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0Î$‡‚H2Ü- _wandbruntime‚4758  _runtime‚4758LJܑÿ&ÝùM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚344999021188096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24889004 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚349691577389056 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4692556200960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76181835 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4693466185728 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85611918 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚344999021188096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24889623 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚385577128492032 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 $ _timestamp‚1.788500700873396e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚344999039191040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚621 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103794311 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4693466185728 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189379857 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76182456 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189406229 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85611916 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 _step‚335 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51292833 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4905932506112 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4675008260138 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349692505376768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103767941 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚338279456995254 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189406229 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚344999039191040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570ŽÔg'ÞþM \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4675916235350 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103794311 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚380671172045824 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189379857 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚333603523720544 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚349692505376768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189406229 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85611916 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 _step‚335 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51292833 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4905932506112 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4675008260138 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349692505376768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103767941 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚338279456995254 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189406229 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚344999039191040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚344999021188096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24889004 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚349691577389056 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4692556200960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76181835 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4693466185728 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85611918 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚344999021188096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24889623 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚385577128492032 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 $ _timestamp‚1.788500700873396e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚344999039191040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚621 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103794311 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4693466185728 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189379857 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76182456 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353745878559232 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77059205 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25175187 % _timestamp‚1.7885007570269508e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚348998356393472 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4814.804293137 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚390055454616064 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51884018 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191587768 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚348998356393472 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86599758 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚353746806533632 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚390055701046784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0v¶~Åû'íõO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4748432137216 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚353746806533632 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4730676216972 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4747522165760 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104988012 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353745878559232 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77059205 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25175187 % _timestamp‚1.7885007570269508e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚348998356393472 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4814.804293137 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚390055454616064 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51884018 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191587768 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚348998356393472 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86599758 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚353746806533632 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚390055701046784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0ÒÑ›[‘î‚(:† •¶éÔ›–¨l gpu.0.gpu‚20.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚184.816' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚46.204 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õa¬°oï‚(:ä •¶éÔÅÔÄm) disk./.usagePercent‚83.77066921121721 disk.sda2.in‚1.671875 disk.md0.in‚ 1.94140625 disk.rbd2.out‚ 1.404296875 disk.sdb2.in‚ 0.26953125! disk.sdb2.out‚531.01806640625 disk.sda.in‚1.671875 disk.sda.out‚531.01806640625! disk.sda2.out‚531.01806640625 disk.nvme0n1.in‚ 105.51953125 disk.sda1.out‚0 disk.rbd0.out‚ 1.404296875' proc.memory.availableMB‚ 657.8359375# proc.memory.rssMB‚ 3011.55078125 cpu‚25.168843925684957 disk.dm-0.out‚58754.57421875 network.sent‚ 9.658268e+06 network.recv‚ 1.623109e+06$ disk./.usageGB‚2996.044895172119 disk.md0p1.out‚ 510.44140625 disk.rbd2.in‚0 disk.sdb.in‚ 0.26953125 disk.md0.out‚ 510.44140625* proc.memory.percent‚18.381047248840332 disk.nvme1n1.in‚ 81.7734375 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.md0p1.in‚ 1.94140625 disk.dm-0.in‚ 187.29296875 disk.sdb.out‚531.01806640625 disk.nvme0n1.out‚ 35863.6875# disk.nvme1n1.out‚22892.13671875 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0$ memory_percent‚95.98488807678223Ü+:2ð- _wandbruntime‚4815  _runtime‚4815´z’8NñúM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25181008 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4731793490458 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191632759 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚353828619105280 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚339 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚337554403447744 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _runtime‚4815.856306037 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚349079993482240 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191632759 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚353829547092992 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚353829547092992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚349079993482240 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4749553610752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105039213 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235060224 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86619916 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105039213 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86619918 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191659131 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4730885515246 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚349079975479296 trainer/global_step‚4149 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25180368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=De_&Ò³²viceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105012843 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚390147101395968 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77076449 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚385182457268224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚342285288962990 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0^•Þ<'òÿM FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77076449 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚385182457268224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚342285288962990 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25181008 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4731793490458 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191632759 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚353828619105280 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚339 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚337554403447744 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _runtime‚4815.856306037 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚349079993482240 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191632759 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚353829547092992 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚353829547092992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚349079993482240 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4749553610752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105039213 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235060224 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86619916 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105039213 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86619918 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191659131 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4730885515246 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚349079975479296 trainer/global_step‚4149 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25180368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105012843 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚390147101395968 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372Ó¨q²p2ó- _wandbruntime‚4815  _runtime‚4815BåùWôÑ 1loss/FBt2/assignment_loss‚0.25130951127026824 /loss/FRt2/detection_loss‚2.7254316920284616 1loss/SRqqt3/detection_loss‚1.0602830438789996 epoch‚3 0loss/FBt1/assignment_loss‚0.2979421094054444 0loss/FBt4/assignment_loss‚0.0916407492380434 _runtime‚4815.857942834 $ _timestamp‚1.788500758080794e+09 /loss/FRt1/detection_loss‚1.7496945949290723 /loss/FBt4/detection_loss‚3.6570155223210654 /loss/FRt1/assignment_loss‚2.971379612882932 2loss/SRqqt2/assignment_loss‚1.8543345374720437 2loss/SRqqt4/assignment_loss‚1.9096696692321338 1loss/SRqqt4/detection_loss‚2.4518542777408254 .loss/FBt2/detection_loss‚2.630052853714336 trainer/global_step‚4149 1loss/SRqqt3/assignment_loss‚1.106615346627818 .loss/FBt3/detection_loss‚2.704438380897045 %loss/total_loss‚46.77207880553999 0loss/SRqqt1/detection_loss‚2.123935865504401 1loss/SRqqt2/detection_loss‚2.7258871580873216 _step‚340 /loss/FRt3/assignment_loss‚1.785277210175991 0loss/FRt4/assignment_loss‚3.6342841585477195 2loss/SRqqt1/assignment_loss‚1.4829958687270326 0loss/FBt3/assignment_loss‚0.4767104727070546 .loss/FRt3/detection_loss‚1.764772665221244 /loss/FBt1/detection_loss‚0.5342224415807003 0loss/FRt2/assignment_loss‚3.8928993344306946 .loss/FRt4/detection_loss‚2.889431728919347MÁ\õÖ epoch‚3 0loss/FBt1/assignment_loss‚0.2979421094054444 0loss/FBt4/assignment_loss‚0.0916407492380434 _runtime‚4815.857942834 $ _timestamp‚1.788500758080794e+09 /loss/FRt1/detection_loss‚1.7496945949290723 /loss/FBt4/detection_loss‚3.6570155223210654 /loss/FRt1/assignment_loss‚2.971379612882932 2loss/SRqqt2/assignment_loss‚1.8543345374720437 2loss/SRqqt4/assignment_loss‚1.9096696692321338 1loss/SRqqt4/detection_loss‚2.4518542777408254 .loss/FBt2/detection_loss‚2.630052853714336 trainer/global_step‚4149 1loss/SRqqt3/assignment_loss‚1.106615346627818 .loss/FBt3/detection_loss‚2.704438380897045 %loss/total_loss‚46.77207880553999 0loss/SRqqt1/detection_loss‚2.123935865504401 1loss/SRqqt2/detection_loss‚2.7258871580873216 _step‚340 /loss/FRt3/assignment_loss‚1.785277210175991 0loss/FRt4/assignment_loss‚3.6342841585477195 2loss/SRqqt1/assignment_loss‚1.4829958687270326 0loss/FBt3/assignment_loss‚0.4767104727070546 .loss/FRt3/detection_loss‚1.764772665221244 /loss/FBt1/detection_loss‚0.5342224415807003 0loss/FRt2/assignment_loss‚3.8928993344306946 .loss/FRt4/detection_loss‚2.889431728919347 1loss/FBt2/assignment_loss‚0.25130951127026824 /loss/FRt2/detection_loss‚2.7254316920284616 1loss/SRqqt3/detection_loss‚1.0602830438789996ÔÛ2Ýähö‚(:Ý ¤¶éÔ™ÁŠn disk.sdb1.out‚0 disk.sda2.out‚532.3583984375 disk.sda.in‚1.671875 disk.sda1.out‚0 disk.dm-0.out‚58960.78515625# proc.memory.rssMB‚ 3011.55078125 cpu‚25.168316448314243 network.sent‚ 9.754381e+06 disk.sda1.in‚0 disk.rbd0.out‚ 1.404296875 disk.sdb.in‚ 0.26953125 disk.nvme1n1.in‚ 81.7734375# memory_percent‚95.9813117980957* proc.memory.percent‚18.381047248840332 network.recv‚ 1.628613e+06# disk.nvme0n1.out‚36068.49609375 disk.md0.in‚ 1.94140625 disk.md0.out‚ 511.734375 disk.rbd0.in‚0 disk.md0p1.in‚ 1.94140625 disk.md0p1.out‚ 511.734375 disk.sda2.in‚1.671875" disk.nvme1n1.out‚ 22892.2890625 disk.rbd2.out‚ 1.404296875& proc.memory.availableMB‚ 658.421875 disk.sdb1.in‚0 disk.dm-0.in‚ 187.29296875 disk.sda.out‚532.3583984375 disk.sdb.out‚532.3583984375 disk.nvme0n1.in‚ 105.51953125 disk.rbd2.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚83.77172995020916 disk.sdb2.out‚532.3583984375$ disk./.usageGB‚2996.082832336426 disk.sdb2.in‚ 0.26953125pÍÊð‘÷‚(:† ¤¶éÔü¿Òp gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<ß|gø‚(:Ü ³¶éÔÞÚ€m network.sent‚ 9.853633e+06 disk.sdb2.in‚ 0.26953125 disk.sda1.in‚0 disk.md0p1.in‚ 1.94140625 disk.md0.out‚ 513.16796875 disk.sda.in‚1.671875$ memory_percent‚95.97995281219482 disk.sdb2.out‚ 533.841796875# proc.memory.rssMB‚ 3011.55078125 disk.rbd2.out‚ 1.404296875 disk.dm-0.out‚ 59135.78125 disk.sdb.out‚ 533.841796875 disk.sda2.in‚1.671875 disk.nvme1n1.in‚ 81.7734375 disk.rbd2.in‚0 disk.sda.out‚ 533.841796875 disk.dm-0.in‚ 187.29296875* proc.memory.percent‚18.381047248840332 proc.cpu.threads‚17 network.recv‚ 1.633687e+06) disk./.usagePercent‚83.74334523607986$ disk./.usageGB‚2995.067657470703 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 1.404296875 disk.sdb.in‚ 0.26953125 disk.md0.in‚ 1.94140625 disk.sda2.out‚ 533.841796875 cpu‚25.167912352077593" disk.nvme1n1.out‚ 22892.4140625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0p1.out‚ 513.16796875 disk.nvme0n1.in‚ 105.51953125" disk.nvme0n1.out‚ 36243.3671875( proc.memory.availableMB‚ 658.64453125â¸üë’ù‚(:‡ ³¶éÔ´äÄ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚144.212' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.053 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07ó"uiú‚(:Þ ¶éÔÐï„m disk.sdb.in‚ 0.26953125 disk.sdb.out‚535.7861328125# disk.nvme1n1.out‚22892.54296875 disk.sdb2.out‚535.7861328125 disk.sda.in‚1.671875 disk.sda2.out‚535.7861328125 disk.sdb1.out‚0 disk.md0p1.in‚ 1.94140625 disk.sda2.in‚1.671875 disk.rbd0.out‚ 1.404296875 disk.sdb2.in‚ 0.26953125( proc.memory.availableMB‚ 658.37890625* proc.memory.percent‚18.381047248840332 proc.cpu.threads‚17 network.sent‚ 9.857809e+06 network.recv‚ 1.638651e+06 cpu‚25.16749875695336 disk.dm-0.in‚ 187.29296875 disk.md0.in‚ 1.94140625$ disk./.usageGB‚2995.278896331787 disk.md0p1.out‚ 515.0390625# disk.nvme0n1.out‚36390.92578125 disk.md0.out‚ 515.0390625 disk.rbd2.out‚ 1.404296875 disk.sda1.out‚0$ memory_percent‚95.98157405853271# proc.memory.rssMB‚ 3011.55078125( disk./.usagePercent‚83.7492515630461 disk.sdb1.in‚0 disk.nvme1n1.in‚ 81.7734375 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-0.out‚ 59283.46875 disk.nvme0n1.in‚ 105.51953125 disk.rbd2.in‚0 disk.sda.out‚535.7861328125'wjO“û‚(:ˆ ¶éÔ¦‰“m gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚163.763' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.94075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·y%2ü- _wandbruntime‚4868  _runtime‚4868ŽBqÆÇýÁ _runtime‚4868.426291244 _step‚341 & lr-AdamW/pg1‚0.0008681641424441672 & lr-AdamW/pg2‚0.0008681641424441672 trainer/global_step‚4199 % _timestamp‚1.7885008106490731e+09n¢ò,ÌþÆ _runtime‚4868.426291244 _step‚341 & lr-AdamW/pg1‚0.0008681641424441672 & lr-AdamW/pg2‚0.0008681641424441672 trainer/global_step‚4199 % _timestamp‚1.7885008106490731e+09Õ¶Àào2ÿ- _wandbruntime‚4868  _runtime‚4868f$Œ”ö'€ðO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106232914 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4803609590784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚394625740180480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87607758 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4786553472080 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106259258 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4785645497076 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚357882920275456 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚353079328687616 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚389603516786176 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4804519562240 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106259258 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52487268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52487266 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4803609590784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚357883848249856 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚4868.427477409 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚357883848249856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4804519562240 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760;SÊY õO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚389603540726272 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393472 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚341426265580400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193867016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5021979060736 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚353079328687616 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77952184 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77952804 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25464918 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106232914 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4803609590784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocatedb é_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚394625740180480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87607758 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4786553472080 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106259258 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4785645497076 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚357882920275456 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚353079328687616 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚389603516786176 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4804519562240 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106259258 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52487268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52487266 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4803609590784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚357883848249856 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚4868.427477409 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚357883848249856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4804519562240 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚357966588809216 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _step‚343 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193885661 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚389693742490624 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4804731051008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5023381036544 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885008115978212e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87627918 _runtime‚4869.375816176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4805641035776 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5023158559232 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25470655 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚394717147467264 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106257745 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77970604 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77969986 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚346292045945298 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87627916 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106284115 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚357966588809216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193912033 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193912033®½:'„ÿM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87627916 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4805641035776 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4804731051008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚353160947773440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193912033 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193912033 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106257745 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚353160947773440 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106284115 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚389693766430720 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚357966588809216 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _step‚343 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193885661 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚389693742490624 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4804731051008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5023381036544 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885008115978212e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87627918 _runtime‚4869.375816176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4805641035776 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5023158559232 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25470655 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚394717147467264 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106257745 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77970604 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77969986 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚346292045945298 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87627916 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106284115 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚357966588809216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚345377162346960 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚345377145307600 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚357160282978816 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4860606987264 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25755959 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53090516 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4859697015808 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107477816 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚357160264975872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107504160 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚399195541449216 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚362019961991680 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4860606987264 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚350218668059784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235944960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88615758 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78846475 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78847112 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 _step‚346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5080739440640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚394114825948672 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4859697015808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚345377162346960 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚345377145307600 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚357160282978816 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4860606987264 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25755959 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53090516 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4859697015808 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107477816 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚357160264975872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107504160 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚399195541449216 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚362019961991680 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4860606987264 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚350218668059784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235944960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88615758 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78846475 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78847112 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 _step‚346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5080739440640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚394114825948672 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4859697015808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0ÚS&S\2–- _wandbruntime‚4923  _runtime‚4923]æ•=—úM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚357241902064640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78864901 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4860818476032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196138563 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107502647 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196164935 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107529017 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚347 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚345456179941504 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4861728460800 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚362102702537728 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚362103630525440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚345456162902144 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4843548000674 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4860818476032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 196138563 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚362103630525440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88635918 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78864273 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚357241884061696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25762318 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚394205051653120 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88635916 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5082127655936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_ ‰ûïà train_batch_end/segment.small_pool.current‚570 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚357241884061696 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚399287179309056 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107502647 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚394205027713024 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196164935 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107529017 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25761692 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4842640025462 _runtime‚4923.961169013 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚350299727942178 trainer/global_step‚4249 % _timestamp‚1.7885008661828756e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88635918 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4861728460800 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5081896790016 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚357241902064640 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0cD‚'˜ÿM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚362102702537728 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚362103630525440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚345456162902144 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4843548000674 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4860818476032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 196138563 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚362103630525440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88635918 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78864273 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚357241884061696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25762318 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚394205051653120 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88635916 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5082127655936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚357241884061696 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚399287179309056 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107502647 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚394205027713024 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196164935 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107529017 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25761692 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4842640025462 _runtime‚4923.961169013 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚350299727942178 trainer/global_step‚4249 % _timestamp‚1.7885008661828756e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88635918 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4861728460800 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5081896790016 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚357241902064640 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚357241902064640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78864901 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4860818476032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196138563 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107502647 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196164935 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107529017 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚347 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚345456179941504 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4861728460800 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212Û¶ŠÞç2™- _wandbruntime‚4923  _runtime‚4923µžZ¼`šÚ 2loss/SRqqt2/assignment_loss‚1.5760202144200985 /loss/FBt2/detection_loss‚2.3316932180348564 epoch‚3 2loss/SRqqt1/assignment_loss‚1.3155092607651437 1loss/SRqqt3/detection_loss‚0.8416778988264981 2loss/SRqqt3/assignment_loss‚0.9476695230471031 1loss/SRqqt2/detection_loss‚2.4904909294385176 _runtime‚4923.963229703 /loss/FRt4/assignment_loss‚3.323072736913508 3loss/FBt3/assignment_loss‚0.0012569625017931686 /loss/FRt1/detection_loss‚1.6689631914047318 /loss/FBt3/detection_loss‚2.5342174410820006 &loss/total_loss‚41.605147363412826 trainer/global_step‚4249 % _timestamp‚1.7885008661860576e+09 2loss/SRqqt4/assignment_loss‚1.6325573985272184 .loss/FRt3/detection_loss‚1.674591111449095 /loss/FBt4/detection_loss‚2.5561548590660097 1loss/FBt1/assignment_loss‚0.05190308967252101 1loss/FBt4/assignment_loss‚0.15728799128070628 1loss/SRqqt1/detection_loss‚2.3594388546688214 1loss/SRqqt4/detection_loss‚2.4037233385546455 /loss/FBt1/detection_loss‚0.8010360873371795 /loss/FRt3/assignment_loss‚1.561295836705428 0loss/FBt2/assignment_loss‚0.2149737255374074 /loss/FRt2/detection_loss‚2.4096213111823257 .loss/FRt4/detection_loss‚3.035991749980233 _step‚348 /loss/FRt1/assignment_loss‚2.819976678159502 0loss/FRt2/assignment_loss‚2.8960239548574793³½e›ß /loss/FRt4/assignment_loss‚3.323072736913508 3loss/FBt3/assignment_loss‚0.0012569625017931686 /loss/FRt1/detection_loss‚1.6689631914047318 /loss/FBt3/detection_loss‚2.5342174410820006 &loss/total_loss‚41.605147363412826 trainer/global_step‚4249 % _timestamp‚1.7885008661860576e+09 2loss/SRqqt4/assignment_loss‚1.6325573985272184 .loss/FRt3/detection_loss‚1.674591111449095 /loss/FBt4/detection_loss‚2.5561548590660097 1loss/FBt1/assignment_loss‚0.05190308967252101 1loss/FBt4/assignment_loss‚0.15728799128070628 1loss/SRqqt1/detection_loss‚2.3594388546688214 1loss/SRqqt4/detection_loss‚2.4037233385546455 /loss/FBt1/detection_loss‚0.8010360873371795 /loss/FRt3/assignment_loss‚1.561295836705428 0loss/FBt2/assignment_loss‚0.2149737255374074 /loss/FRt2/detection_loss‚2.4096213111823257 .loss/FRt4/detection_loss‚3.035991749980233 _step‚348 /loss/FRt1/assignment_loss‚2.819976678159502 0loss/FRt2/assignment_loss‚2.8960239548574793 2loss/SRqqt2/assignment_loss‚1.5760202144200985 /loss/FBt2/detection_loss‚2.3316932180348564 epoch‚3 2loss/SRqqt1/assignment_loss‚1.3155092607651437 1loss/SRqqt3/detection_loss‚0.8416778988264981 2loss/SRqqt3/assignment_loss‚0.9476695230471031 1loss/SRqqt2/detection_loss‚2.4904909294385176 _runtime‚4923.963229703ÜW‘’œ‚(:‡ ·éÔñ‡‚n gpu.0.gpu‚32.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.774' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.9435 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0IfÙo‚(:ä ·éÔääÛn" disk.nvme0n1.out‚ 37046.0078125 disk.rbd0.in‚0 disk.dm-0.in‚ 187.29296875* proc.memory.percent‚18.381047248840332" disk.nvme1n1.out‚ 22893.8515625 disk.md0.in‚ 1.94140625% disk./.usageGB‚2995.2039375305176 disk.nvme0n1.in‚ 105.51953125 disk.sdb2.in‚ 0.26953125 disk.sda.out‚543.65283203125 disk.sdb1.in‚0 disk.dm-0.out‚ 59939.859375 cpu‚25.165301099478953 disk.rbd0.out‚ 1.404296875# memory_percent‚95.9843397140503 network.recv‚ 1.659745e+06) disk./.usagePercent‚83.74715568358991 disk.md0.out‚ 522.63671875 disk.rbd2.in‚0! disk.sdb2.out‚543.65283203125 disk.sda1.in‚0# proc.memory.rssMB‚ 3011.55078125 proc.cpu.threads‚17 disk.md0p1.in‚ 1.94140625 network.sent‚ 9.98712e+06 disk.sdb.out‚543.65283203125! disk.sda2.out‚543.65283203125 disk.rbd2.out‚ 1.404296875 disk.sda.in‚1.671875 disk.sdb1.out‚0 disk.sdb.in‚ 0.26953125 disk.sda1.out‚0 disk.md0p1.out‚ 522.63671875( proc.memory.availableMB‚ 657.92578125 disk.sda2.in‚1.671875 disk.nvme1n1.in‚ 81.7734375ó5¤3“ž‚(:ˆ œ·éÔ•¯öl gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.219' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.55475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±+¨hŸ‚(:Ý œ·éÔ„ŠÐm network.recv‚ 1.665461e+06# proc.memory.rssMB‚ 3011.55078125 cpu‚25.16486455496647 disk.dm-0.out‚ 60068.03125 disk.sdb.in‚ 0.26953125 disk.sdb.out‚545.47119140625 disk.nvme1n1.in‚ 81.7734375 disk.nvme1n1.out‚ 22893.96875( disk./.usagePercent‚83.7586004653471 disk.sda1.out‚0 disk.rbd0.out‚ 1.404296875 disk.sda2.in‚1.671875 disk.rbd0.in‚0 disk.md0p1.out‚ 524.3984375 disk.nvme0n1.out‚ 37174.0625 proc.cpu.threads‚17 disk.dm-0.in‚ 187.29296875 disk.md0.out‚ 524.3984375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.out‚ 1.404296875 disk.sdb2.in‚ 0.26953125 disk.sda.in‚1.671875$ memory_percent‚95.98281383514404 network.sent‚ 1.0096973e+07 disk.md0p1.in‚ 1.94140625! disk.sda2.out‚545.47119140625 disk.nvme0n1.in‚ 105.51953125! disk.sdb2.out‚545.47119140625 disk.sda.out‚545.47119140625% disk./.usageGB‚2995.6132583618164 disk.sda1.in‚0 disk.md0.in‚ 1.94140625 disk.rbd2.in‚0( proc.memory.availableMB‚ 658.17578125* proc.memory.percent‚18.381047248840332à:KË“ ‚(:ˆ «·éÔØÊ‚m gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.333' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.33325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿ll¡‚(:á «·éÔÐ͸n disk.sdb2.in‚ 0.26953125 disk.sda.out‚547.38720703125 disk.sda2.in‚1.671875* proc.memory.percent‚18.381047248840332 network.sent‚ 1.0102874e+07 disk.sda1.out‚0 disk.sdb1.out‚0 disk.md0p1.out‚ 526.24609375 disk.md0.in‚ 1.94140625# proc.memory.rssMB‚ 3011.55078125! disk.sdb2.out‚547.38720703125 disk.rbd0.in‚0 disk.md0p1.in‚ 1.94140625 disk.dm-0.in‚ 187.29296875 disk.sdb.in‚ 0.26953125 disk.nvme1n1.in‚ 81.7734375' proc.memory.availableMB‚ 658.1796875 cpu‚25.16446087636079 network.recv‚ 1.66938e+06 disk.dm-0.out‚ 60177.515625! disk.sda2.out‚547.38720703125 disk.nvme0n1.in‚ 105.51953125 disk.rbd2.in‚0$ memory_percent‚95.98278999328613 proc.cpu.threads‚17 disk.sda.in‚1.671875 disk.sda1.in‚0 disk.sdb1.in‚0 disk.nvme0n1.out‚ 37283.40625) disk./.usagePercent‚83.76470272771085% disk./.usageGB‚2995.8315048217773 disk.rbd0.out‚ 1.404296875 disk.sdb.out‚547.38720703125! disk.nvme1n1.out‚ 22894.109375 disk.md0.out‚ 526.24609375 disk.rbd2.out‚ 1.404296875ZÈ…92¢- _wandbruntime‚4979  _runtime‚4979]MæÇ£Á % _timestamp‚1.7885009219344594e+09 _runtime‚4979.711719379 _step‚349 & lr-AdamW/pg1‚0.0008564776672106018 & lr-AdamW/pg2‚0.0008564776672106018 trainer/global_step‚4299˧?Ì¤Æ & lr-AdamW/pg1‚0.0008564776672106018 & lr-AdamW/pg2‚0.0008564776672106018 trainer/global_step‚4299 % _timestamp‚1.7885009219344594e+09 _runtime‚4979.711719379 _step‚349Ý¡ ï#2¥- _wandbruntime‚4979  _runtime‚4979W0ØGô'¦îO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79740804 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4915784440832 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89623758 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚361241237270016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198372820 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4897400007292 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚366157931682304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53693766 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89623758 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚361241219267072 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚403765762898944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚403765516468224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚398626087230976 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _runtime‚4979.712862664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79741427 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4916694412288 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89623756 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5139429237248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26047659 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4915784440832 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4898307982296 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53693768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚366157931682304 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366157003707904 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248¥ßå¾ä§óO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89623756 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108749062 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚398626111171072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚354225425042092 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366157003707904 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79740804 OIDeviceStatsMonitor.on_train_batch_stìÊÝr art/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4915784440832 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89623758 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚361241237270016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198372820 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4897400007292 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚366157931682304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53693766 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89623758 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚361241219267072 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚403765762898944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚403765516468224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚398626087230976 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _runtime‚4979.712862664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79741427 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4916694412288 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89623756 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5139429237248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26047659 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4915784440832 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4898307982296 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53693768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚366157931682304 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚398716312935424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚361322856355840 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚349407042629344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198391465 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚403856925058560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216185856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4899425255782 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚354306484924486 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚366240672241664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚361322838352896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26052820 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198391465 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89643916 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240125952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4916905901056 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4898517280570 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89643918 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108747549 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198417837 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108773919 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4917815885824 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79758651 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53705833 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _step‚351 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚361322856355840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53705831 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚398716336875520 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 trainer/global_step‚4299 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚631 _runtime‚4980.738752136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4917815885824 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚354305559909914 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4916905901056 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89643918 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108773919 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5140612123136 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚403857165184512-³’'¬ÿM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚398716312935424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚361322856355840 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚349407042629344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198391465 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚403856925058560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216185856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4899425255782 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚354306484924486 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚366240672241664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚361322838352896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26052820 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198391465 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89643916 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240125952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4916905901056 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4898517280570 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89643918 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108747549 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198417837 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108773919 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4917815885824 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79758651 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53705833 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _step‚351 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚361322856355840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53705831 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚398716336875520 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 trainer/global_step‚4299 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚631 _runtime‚4980.738752136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4917815885824 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚354305559909914 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4916905901056 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89643918 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108773919 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5140612123136 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚403857165184512 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚349407059668704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 % _timestamp‚1.7885009229606729e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249ߢåÈ2­- _wandbruntime‚4980  _runtime‚4980W YÙ^®Ø 2loss/SRqqt3/assignment_loss‚1.3250265829998442 0loss/SRqqt3/detection_loss‚0.915674637709781 .loss/FBt4/detection_loss‚2.379954559462411 /loss/FBt1/detection_loss‚0.8269987358042682 1loss/SRqqt1/detection_loss‚2.1736439367135367 trainer/global_step‚4299 /loss/FRt1/detection_loss‚1.7985067883071715 .loss/FRt2/detection_loss‚2.516519546704857 &loss/total_loss‚44.056601533862434 _runtime‚4980.740929796 /loss/FBt3/detection_loss‚1.7258380502462387 epoch‚3 0loss/FRt2/assignment_loss‚3.3269850611686707 0loss/FRt3/assignment_loss‚1.8209152945450373 1loss/FBt1/assignment_loss‚0.11387481322630602 0loss/FBt3/assignment_loss‚0.1862541680351504 /loss/FRt3/detection_loss‚2.0558739514755353 1loss/SRqqt2/detection_loss‚2.6061928065923543 0loss/FRt1/assignment_loss‚2.2496245783918045 2loss/SRqqt1/assignment_loss‚1.6359591698274016 2loss/SRqqt2/assignment_loss‚1.5882966696069791 2loss/SRqqt4/assignment_loss‚1.6060449494556948 1loss/FBt2/assignment_loss‚0.38020847164298177 0loss/FBt4/assignment_loss‚0.6035168671839658 1loss/SRqqt4/detection_loss‚2.7164992961016567 /loss/FBt2/detection_loss‚2.3199936230977376 /loss/FRt4/assignment_loss‚4.128822160618646 /loss/FRt4/detection_loss‚3.0553768149444034 % _timestamp‚1.7885009229636896e+09 _step‚352ÃcDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚408335405913088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54297016 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90631758 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚408335151093760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26338550 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚403137396393472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 trainer/global_step‚4349 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚365322173558272 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚353278921801360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4953277262400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚353278904762000 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚370294045424128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚638 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5198009519616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚358232182024400 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90631758 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚370294973398528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4972781837312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 % _timestamp‚1.7885009775287743e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚403137372453376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚370294045424128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚365322191561216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4971871865856 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚370294973398528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109993964 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4954185237404 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26337912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚358233107038764 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109967620 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90631756 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4972781837312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚365322191561216 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200625722 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570(Ì(û'»õO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚408335405913088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54297016 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90631758 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚408335151093760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26338550 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚403137396393472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 trainer/global_step‚4349 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚365322173558272 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚353278921801360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4953277262400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚353278904762000 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚370294045424128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚638 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5198009519616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚358232182024400 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90631758 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚370294973398528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4972781837312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 % _timestamp‚1.7885009775287743e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚403137372453376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚370294045424128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚365322191561216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4971871865856 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚370294973398528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109993964 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4954185237404 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26337912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚358233107038764 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109967620 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90631756 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4972781837312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚365322191561216 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200625722 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90631756 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006â Ï/2¼- _wandbruntime‚5036  _runtime‚5036†i$W½úM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54309081 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200644367 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚625 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚408426549961216 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4954394535678 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4973903310848 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4972993326080 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚365403810647040 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4972993326080 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200670739 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚627 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 _step‚355 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4973903310848 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚370376785970176 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚370377713957888 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403227622097920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚358312316892222 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110018821 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚365403810647040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80652847 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26344391 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109992451 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚365403792644096 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26343766 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 _runtime‚5036.274553423 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚353357922356544 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90651918 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90651916 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109992451 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5199174280704 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2lÆ'¾ÿM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚353357939395904 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚403227598157824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90651918 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5198951803392 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚358313241906794 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚625 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚408426549961216 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4954394535678 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4973903310848 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4972993326080 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚365403810647040 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4972993326080 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200670739 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚627 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 _step‚355 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4973903310848 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚370376785970176 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚370377713957888 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403227622097920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚358312316892222 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110018821 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚365403810647040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80652847 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26344391 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109992451 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚365403792644096 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26343766 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 _runtime‚5036.274553423 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚353357922356544 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90651918 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90651916 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109992451 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5199174280704 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54309081 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200644367 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5256683389952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚622 % _timestamp‚1.7885010330657291e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5010062492512 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5027959290880 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5028869262336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚412905365005824 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5256454607872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 trainer/global_step‚4399 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5028869262336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5027959290880 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91639756 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚374432015114752 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91639758 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26628495 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202852278 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚374432015114752 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81528761 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26629115 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚369403127849472 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚369403127849472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚362238939006708 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202878624 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81529383 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374431087140352 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111212522 _step‚358 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111212522 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91639756 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚374431087140352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54900266 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5009154517508 _runtime‚5090.843503013 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚369403145852416 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚407648657675776 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚407648681615872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚412905112283648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111238866 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54900268 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚357229784489200 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚369403145852416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚357229801528560 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91639758 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111238866 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚362239864021072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185‹·ÏõO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91639758 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_  à&train_batch_start/allocation.small_pool.allocated‚ 111238866 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚362239864021072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5256683389952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚622 % _timestamp‚1.7885010330657291e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5010062492512 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5027959290880 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5028869262336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚412905365005824 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5256454607872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 trainer/global_step‚4399 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5028869262336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5027959290880 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91639756 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚374432015114752 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91639758 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26628495 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202852278 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚374432015114752 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81528761 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26629115 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚369403127849472 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚369403127849472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚362238939006708 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202878624 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81529383 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374431087140352 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111212522 _step‚358 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111212522 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91639756 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚374431087140352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54900266 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5009154517508 _runtime‚5090.843503013 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚369403145852416 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚407648657675776 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚407648681615872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚412905112283648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111238866 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54900268 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚357229784489200 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚369403145852416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚357229801528560 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0挠Ý2Ð- _wandbruntime‚5091  _runtime‚5091B.A˜'ÑúM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202897269 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚412996483822592 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚374513827686400 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202923641 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54912331 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚362319998889102 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202923641 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚407738907320320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 239254528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚374514755674112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111237353 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91659916 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 263194624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5257600442368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚369484746935296 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91659918 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5011179765998 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4399 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚362319073874530 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91659916 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91659918 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5029990735872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26634254 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚369484764938240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111263723 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81546585 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚357308819123104 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚5091.946372448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111237353 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111263723 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚374514755674112 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚369484764938240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26634878ÁpIÑ'ÒÿM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚369484746935296 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91659918 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5011179765998 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4399 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚362319073874530 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91659916 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91659918 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5029990735872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26634254 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚369484764938240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111263723 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81546585 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚357308819123104 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚5091.946372448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111237353 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111263723 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚374514755674112 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚369484764938240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26634878 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202897269 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚412996483822592 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚374513827686400 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202923641 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54912331 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚362319998889102 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202923641 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚407738907320320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 239254528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚374514755674112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111237353 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91659916 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 263194624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5257600442368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760çøu9—2Ó- _wandbruntime‚5091  _runtime‚5091tÉ$ó[ÔÕ /loss/FRt2/assignment_loss‚3.841687617983137 2loss/SRqqt1/assignment_loss‚1.9056472486005194 1loss/FBt1/assignment_loss‚0.18137329480282074 /loss/FRt1/detection_loss‚1.8769182354095393 1loss/SRqqt3/detection_loss‚0.7998267979391159 /loss/FBt2/detection_loss‚2.5682281891504926 _step‚360 0loss/FRt4/assignment_loss‚3.2543440965505748 1loss/FBt2/assignment_loss‚0.32371825956227746 0loss/FBt3/assignment_loss‚0.5397642984254551 0loss/SRqqt1/detection_loss‚1.746475491792925 epoch‚3 trainer/global_step‚4399 0loss/FRt1/assignment_loss‚2.6908462206522623 1loss/SRqqt2/assignment_loss‚1.592335679150861 1loss/FBt4/assignment_loss‚0.24009399510703133 0loss/SRqqt2/detection_loss‚2.436887239587718 %loss/total_loss‚43.74316905977105 1loss/SRqqt3/assignment_loss‚1.091374866521707 .loss/FRt3/detection_loss‚1.631539327997557 /loss/FRt4/detection_loss‚3.0788405400056105 % _timestamp‚1.7885010341704798e+09 _runtime‚5091.947618834 2loss/SRqqt4/assignment_loss‚1.8919898110628128 0loss/SRqqt4/detection_loss‚2.596769616603851 /loss/FBt3/detection_loss‚2.2607558029038564 /loss/FBt4/detection_loss‚2.3917006738483906 0loss/FRt3/assignment_loss‚1.6313898252769254 /loss/FRt2/detection_loss‚2.4356593114989145 /loss/FBt1/detection_loss‚0.7350026193366906Ø"2°+ÕÚ /loss/FRt1/detection_loss‚1.8769182354095393 1loss/SRqqt3/detection_loss‚0.7998267979391159 /loss/FBt2/detection_loss‚2.5682281891504926 _step‚360 0loss/FRt4/assignment_loss‚3.2543440965505748 1loss/FBt2/assignment_loss‚0.32371825956227746 0loss/FBt3/assignment_loss‚0.5397642984254551 0loss/SRqqt1/detection_loss‚1.746475491792925 epoch‚3 trainer/global_step‚4399 0loss/FRt1/assignment_loss‚2.6908462206522623 1loss/SRqqt2/assignment_loss‚1.592335679150861 1loss/FBt4/assignment_loss‚0.24009399510703133 0loss/SRqqt2/detection_loss‚2.436887239587718 %loss/total_loss‚43.74316905977105 1loss/SRqqt3/assignment_loss‚1.091374866521707 .loss/FRt3/detection_loss‚1.631539327997557 /loss/FRt4/detection_loss‚3.0788405400056105 % _timestamp‚1.7885010341704798e+09 _runtime‚5091.947618834 2loss/SRqqt4/assignment_loss‚1.8919898110628128 0loss/SRqqt4/detection_loss‚2.596769616603851 /loss/FBt3/detection_loss‚2.2607558029038564 /loss/FBt4/detection_loss‚2.3917006738483906 0loss/FRt3/assignment_loss‚1.6313898252769254 /loss/FRt2/detection_loss‚2.4356593114989145 /loss/FBt1/detection_loss‚0.7350026193366906 /loss/FRt2/assignment_loss‚3.841687617983137 2loss/SRqqt1/assignment_loss‚1.9056472486005194 1ê‚HC5loss/FBt1/assignment_loss‚0.18137329480282074èa¥ ¯dÖ‚(:Ù ²¸éÔº½·o cpu‚25.160996761361115) disk./.usagePercent‚84.07337265466289 disk.nvme1n1.in‚ 81.8984375# disk.nvme1n1.out‚29161.56640625 disk.sdb.out‚565.1103515625 disk.sda2.in‚1.875" proc.memory.rssMB‚ 3011.5546875 network.sent‚ 1.043667e+07 disk.sdb2.in‚ 0.41015625 disk.sda1.in‚0 disk.rbd0.out‚ 1.404296875 disk.sdb1.out‚0 disk.dm-0.in‚ 187.859375 proc.cpu.threads‚17 disk.md0.in‚ 2.28515625# disk.nvme0n1.out‚40728.69140625 disk.rbd0.in‚0# memory_percent‚95.9827184677124 disk.nvme0n1.in‚ 105.9609375 disk.rbd2.in‚0 disk.sda2.out‚565.1103515625 disk.dm-0.out‚ 69890.2578125 disk.sdb.in‚ 0.41015625 network.recv‚ 1.710551e+06 disk.sda.out‚565.1103515625 disk.sdb1.in‚0 disk.md0p1.in‚ 2.28515625 disk.md0p1.out‚ 543.40234375 disk.sdb2.out‚565.1103515625$ disk./.usageGB‚3006.871036529541 disk.md0.out‚ 543.40234375 disk.sda.in‚1.875 disk.rbd2.out‚ 1.404296875 disk.sda1.out‚0( proc.memory.availableMB‚ 658.19140625* proc.memory.percent‚18.381071090698242j? ’ׂ(:‡ ²¸éÔßí¿n gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”딹‘Ø‚(:† Á¸éÔ¦©Îk gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚164.372' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.093 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–X^Ù‚(:Ó Á¸éÔà×n disk.sda2.out‚568.2529296875 disk.rbd2.out‚ 1.404296875 disk.sdb2.in‚ 0.6328125 disk.sdb1.out‚0 disk.md0p1.out‚ 546.4765625 disk.dm-0.in‚ 203.9453125% disk./.usageGB‚3007.6305503845215! disk.nvme1n1.out‚ 30226.140625 disk.sda.in‚3.546875 disk.rbd0.in‚0$ memory_percent‚95.98398208618164 network.sent‚ 1.0545672e+07 network.recv‚ 1.715005e+06 disk.nvme0n1.in‚ 106.5390625 disk.md0.out‚ 546.4765625 proc.cpu.threads‚17 disk.nvme1n1.in‚97.40625 disk.sdb.out‚568.2529296875) disk./.usagePercent‚84.09460897992939! disk.nvme0n1.out‚ 41848.953125 disk.sdb.in‚ 0.6328125 disk.sda2.in‚3.546875" proc.memory.rssMB‚ 3011.5546875 cpu‚25.160437188592034 disk.sda1.in‚0 disk.rbd0.out‚ 1.404296875& proc.memory.availableMB‚ 657.984375 disk.md0.in‚ 4.1796875 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sdb1.in‚0 disk.md0p1.in‚ 4.1796875* proc.memory.percent‚18.381071090698242 disk.sdb2.out‚568.2529296875 disk.sda.out‚568.2529296875 disk.dm-0.out‚ 72075.09375ô~meÚ‚(:„ иéÔȳ¶m gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚157.52' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.38 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þo'+aÛ‚(:Ö иéÔ…¿ðn disk.sdb2.out‚ 570.826171875 disk.sda2.in‚3.546875# disk.nvme0n1.out‚42993.97265625$ disk./.usageGB‚3007.596176147461 disk.nvme1n1.in‚ 98.70703125# disk.nvme1n1.out‚30408.94140625 disk.sda1.in‚0* proc.memory.percent‚18.381071090698242 cpu‚25.1600169543999 disk.rbd2.in‚0 disk.sda.out‚ 570.826171875 disk.sdb1.in‚0 disk.nvme0n1.in‚ 106.78515625 disk.rbd0.in‚0# memory_percent‚95.9824800491333 disk.md0.out‚ 548.9765625 disk.sdb1.out‚0 disk.dm-0.out‚ 73400.2890625 disk.sdb.out‚ 570.826171875" proc.memory.rssMB‚ 3011.5546875 proc.cpu.threads‚17 network.sent‚ 1.0551801e+07 network.recv‚ 1.71969e+06) disk./.usagePercent‚84.09364786187444 disk.rbd2.out‚ 1.404296875 disk.rbd0.out‚ 1.404296875 disk.md0p1.in‚ 4.1796875 disk.sda2.out‚ 570.826171875( proc.memory.availableMB‚ 658.23046875 disk.md0p1.out‚ 548.9765625 disk.sdb.in‚ 0.6328125 disk.sdb2.in‚ 0.6328125 disk.sda.in‚3.546875 disk.md0.in‚ 4.1796875 disk.sda1.out‚0 disk.dm-0.in‚ 205.4921875JEúç‘Ü‚(:† ߸éÔÒ¢m gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.61' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕÜÅeÝ‚(:Ú ߸éÔ¢ÝÔm% disk./.usageGB‚3006.6909408569336 disk.rbd0.in‚0 disk.rbd0.out‚ 1.404296875 disk.sda2.in‚3.546875! disk.nvme1n1.out‚ 31216.984375$ memory_percent‚95.98548412322998 disk.md0.out‚ 550.390625 disk.sdb1.in‚0 disk.dm-0.in‚ 205.4921875 disk.dm-0.out‚ 74774.3515625 disk.sdb.out‚572.3115234375 disk.nvme0n1.in‚ 106.78515625 disk.sda.in‚3.546875 disk.md0p1.in‚ 4.1796875 disk.md0p1.out‚ 550.390625 proc.cpu.threads‚17 disk.sda1.in‚0 network.recv‚ 1.723689e+06 disk.md0.in‚ 4.1796875 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 1.404296875 disk.sdb2.in‚ 0.6328125 disk.sdb1.out‚0 disk.sdb.in‚ 0.6328125 disk.sda2.out‚572.3115234375" proc.memory.rssMB‚ 3011.5546875* proc.memory.percent‚18.381071090698242 disk.sda.out‚572.3115234375" disk.nvme0n1.out‚ 43557.3671875 disk.nvme1n1.in‚ 98.70703125( proc.memory.availableMB‚ 657.73828125 network.sent‚ 1.0557589e+07) disk./.usagePercent‚84.06833710428089 disk.sdb2.out‚572.3115234375 cpu‚25.159516607051465Ã*€c2Þ- _wandbruntime‚5148  _runtime‚5148]2¢Åß¿ _step‚361 % lr-AdamW/pg1‚0.000838434831945227 % lr-AdamW/pg2‚0.000838434831945227 trainer/global_step‚4449 % _timestamp‚1.7885010909792323e+09 _runtime‚5148.756617487V™AjÊàÄ % lr-AdamW/pg1‚0.000838434831945227 % lr-AdamW/pg2‚0.000838434831945227 trainer/global_step‚4449 % _timestamp‚1.7885010909792323e+09 _runtime‚5148.756617487 _step‚361éà伪2á- _wandbruntime‚5148  _runtime‚5148dî hö'âðO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112457424 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5065939747620 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55503516 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚361180681255760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92647758 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚412159966838272 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235073536 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5084956687360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92647756 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5315250230784 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚366245695989016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82421896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚417474958055424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚5148.757912513 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚378568128856576 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚373484100143616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55503518 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26918380 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885010909802945e+09 _step‚362 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚378569056830976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5315015157248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚373484100143616 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚373484082140672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259013632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112483768 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92647758 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26919031 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚378568128856576 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112457424 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5084956687360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5084046715904 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚366246621003380 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5065031772616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0–Å}Èû'ãõO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚417474958055424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚5148.757912513 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚378568128856576 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚373484100143616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55503518 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26918380 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885010909802945e+09 _step‚362 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚378569056830976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5315015157248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚373484100143616 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚373484082140672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259013632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112483768 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92647758 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26919031 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚378568128856576 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112457424 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5084956687360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5084046715904 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚366246621003380 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5065031772616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112457424 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5065939747620 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55503516 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚361180681255760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92647758 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚412159966838272 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235073536 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5084956687360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92647756 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5315250230784 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚366245695989016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82421896ê¬vwg2ä- _wandbruntime‚5149  _runtime‚5149+’é=såùM ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112482255 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205176543 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112508625 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5085168176128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚412250168602624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112508625 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚643 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205150171 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚366325830856838 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26924132 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5316412580352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205176543 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92667916 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚378651797390336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 _step‚363 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 _runtime‚5149.797145729 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92667918 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55515581 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26924775 $ _timestamp‚1.788501092019289e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82439713 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5316179617280 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚361259698850304 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 trainer/global_step‚4449 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚373565701226496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚378651797390336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚373565701226496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5066149045894 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚373565719229440Ä‘ëR'æþM VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205150171 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚366325830856838 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26924132 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5316412580352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205176543 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92667916 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚378651797390336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 _step‚363 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 _runtime‚5149.797145729 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92667918 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55515581 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26924775 $ _timestamp‚1.788501092019289e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82439713 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5316179617280 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚361259698850304 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 trainer/global_step‚4449 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚373565701226496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚378651797390336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚373565701226496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5066149045894 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚373565719229440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112482255 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205176543 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112508625 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5085168176128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚412250168602624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112508625 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚643 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ëØî-2ç- _wandbruntime‚5149  _runtime‚5149´4Û¨ZèÔ 2loss/SRqqt4/assignment_loss‚1.5317539729710137 /loss/FRt4/detection_loss‚3.5650622993707657 $ _timestamp‚1.788501092022111e+09 0loss/FRt3/assignment_loss‚1.4006756749004126 2loss/SRqqt2/assignment_loss‚1.6272524818778038 /loss/FRt3/detection_loss‚1.5995503165759146 /loss/FBt2/detection_loss‚2.1738181710243225 _step‚364 0loss/FRt2/assignment_loss‚3.1735725868039015 0loss/FRt4/assignment_loss‚3.0504622608423233 0loss/FBt2/assignment_loss‚0.1502129349822513 /loss/FBt1/detection_loss‚0.5400444647600223 0loss/FRt1/assignment_loss‚2.4329000038129314 1loss/FBt1/assignment_loss‚0.12972246515064317 /loss/FBt3/assignment_loss‚0.068915058625862 0loss/FBt4/assignment_loss‚0.6201427331624372 0loss/SRqqt2/detection_loss‚2.517692725573267 1loss/SRqqt4/detection_loss‚2.6409778829131807 /loss/FRt1/detection_loss‚1.6940192527103204 2loss/SRqqt1/assignment_loss‚1.0906357095088988 .loss/FRt2/detection_loss‚2.454075002906526 1loss/SRqqt3/detection_loss‚1.0555854285401958 trainer/global_step‚4449 2loss/SRqqt3/assignment_loss‚1.1600279979908668 .loss/FBt3/detection_loss‚3.274106591939926 %loss/total_loss‚42.69992338411981 1loss/SRqqt1/detection_loss‚2.1586368708383468 .loss/FBt4/detection_loss‚2.590080496337679 epoch‚3 _runtime‚5149.799101978zö¸º_éÙ 1loss/SRqqt1/detection_loss‚2.1586368708383468 .loss/FBt4/detection_loss‚2.590080496337679 epoch‚3 _runtime‚5149.799101978 2loss/SRqqt4/assignment_loss‚1.5317539729710137 /loss/FRt4/detection_loss‚3.5650622993707657 $ _timestamp‚1.788501092022111e+09 0loss/FRt3/assignment_loss‚1.4006756749004126 2loss/SRqqt2/assignment_loss‚1.6272524818778038 /loss/FRt3/detection_loss‚1.5995503165759146 /loss/FBt2/detection_loss‚2.1738181710243225 _step‚364 0loss/FRt2/assignment_loss‚3.1735725868039015 0loss/FRt4/assignment_loss‚3.0504622608423233 0loss/FBt2/assignment_loss‚0.1502129349822513 /loss/FBt1/detection_loss‚0.5400444647600223 0loss/FRt1/assignment_loss‚2.4329000038129314 1loss/FBt1/assignment_loss‚0.12972246515064317 /loss/FBt3/assignment_loss‚0.068915058625862 0loss/FBt4/assignment_loss‚0.6201427331624372 0loss/SRqqt2/detection_loss‚2.517692725573267 1loss/SRqqt4/detection_loss‚2.6409778829131807 /loss/FRt1/detection_loss‚1.6940192527103204 2loss/SRqqt1/assignment_loss‚1.0906357095088988 .loss/FRt2/detection_loss‚2.454075002906526 1loss/SRqqt3/detection_loss‚1.0555854285401958 trainer/global_step‚4449 2loss/SRqqt3/assignment_loss‚1.1600279979908668 .loss/FBt3/detection_loss‚3.274106591939926 %loss/total_loss‚42.69992338411981ìa(`“ê‚(:ˆ î¸éÔÒºÑm gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚129.005' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.25125 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08Cájë‚(:ß î¸éÔ°¬©n$ disk./.usageGB‚3006.646797180176 disk.sda2.in‚3.546875 disk.sdb2.in‚ 0.6328125) disk./.usagePercent‚84.06710282860318 disk.sdb.out‚573.8818359375 disk.sda2.out‚573.8818359375 disk.sdb2.out‚573.8818359375 disk.sda.out‚573.8818359375 network.sent‚ 1.0561807e+07 disk.sdb1.in‚0 disk.md0p1.in‚ 4.1796875 disk.dm-0.out‚ 75028.2421875 disk.md0.in‚ 4.1796875 disk.md0.out‚ 551.89453125 disk.rbd2.in‚0 disk.rbd2.out‚ 1.404296875 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 1.404296875" proc.memory.rssMB‚ 3011.5546875 cpu‚25.15919307648692 disk.nvme1n1.in‚ 98.70703125# disk.nvme1n1.out‚31261.17578125 disk.sda.in‚3.546875 disk.md0p1.out‚ 551.89453125 disk.sdb.in‚ 0.6328125# disk.nvme0n1.out‚43767.06640625( proc.memory.availableMB‚ 657.94921875* proc.memory.percent‚18.381071090698242 network.recv‚ 1.728713e+06 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.dm-0.in‚ 205.4921875 disk.nvme0n1.in‚ 106.78515625 disk.sda1.out‚0$ memory_percent‚95.98419666290283JÆÜÇbì‚(:× ý¸éÔä—àm% disk./.usageGB‚3006.7171363830566 disk.rbd2.out‚ 1.404296875 disk.sdb1.out‚0$ memory_percent‚95.98441123962402 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.sdb.in‚ 0.6328125" proc.memory.rssMB‚ 3011.5546875 proc.cpu.threads‚17) disk./.usagePercent‚84.06906954215498 disk.nvme0n1.in‚ 106.78515625! disk.nvme0n1.out‚ 44073.859375 disk.sda1.in‚0 disk.md0p1.out‚ 553.8359375* proc.memory.percent‚18.381071090698242 cpu‚25.158787781454897 disk.sda2.out‚ 575.880859375 disk.rbd0.in‚0 network.sent‚ 1.0667889e+07 disk.sda2.in‚3.546875 disk.sdb2.out‚ 575.880859375 disk.sda.in‚3.546875 network.recv‚ 1.732835e+06 disk.nvme1n1.in‚ 98.70703125 disk.sdb2.in‚ 0.6328125 disk.sda.out‚ 575.880859375" disk.nvme1n1.out‚ 31275.9296875 disk.rbd0.out‚ 1.404296875 disk.md0p1.in‚ 4.1796875 disk.md0.in‚ 4.1796875 disk.md0.out‚ 553.8359375 disk.sda1.out‚0 disk.dm-0.in‚ 205.4921875 disk.dm-0.out‚ 75349.7890625 disk.sdb.out‚ 575.880859375' proc.memory.availableMB‚ 657.9140625).kí‚(:… ý¸éÔÄÆ°o gpu.0.gpu‚5.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚120.65' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.1625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ'5Ÿeî‚(:Ú Œ¹éÔö—èl) disk./.usagePercent‚84.07088138461413 disk.sda1.out‚0 disk.sdb1.in‚0 disk.sda2.out‚ 577.373046875# disk.nvme0n1.out‚44076.08203125" disk.nvme1n1.out‚ 31276.0390625 disk.md0.in‚ 4.1796875 disk.rbd2.out‚ 1.404296875 disk.sda2.in‚3.546875( proc.memory.availableMB‚ 657.42578125" proc.memory.rssMB‚ 3011.5546875 proc.cpu.threads‚17 network.sent‚ 1.067389e+07 disk.sdb2.out‚ 577.373046875 disk.md0p1.out‚ 555.28515625 disk.nvme1n1.in‚ 98.70703125# memory_percent‚95.9873914718628 disk.sdb1.out‚0 disk.dm-0.out‚75352.12109375 disk.sdb.in‚ 0.6328125$ disk./.usageGB‚3006.781936645508 disk.sdb2.in‚ 0.6328125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 205.4921875 disk.rbd2.in‚0* proc.memory.percent‚18.381071090698242 network.recv‚ 1.737172e+06 disk.md0p1.in‚ 4.1796875 disk.sdb.out‚ 577.373046875 disk.sda.in‚3.546875 disk.rbd0.out‚ 1.404296875 disk.nvme0n1.in‚ 106.78515625 cpu‚25.158438078213766 disk.sda.out‚ 577.373046875 disk.md0.out‚ 555.28515625ýš«È’ï‚(:‡ Œ¹éÔ«¶øn gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚174.286' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.5715 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è´p*2ð- _wandbruntime‚5204  _runtime‚5204œfø/ÆñÀ & lr-AdamW/pg2‚0.0008322887202429627 trainer/global_step‚4499 $ _timestamp‚1.788501147191166e+09 _runtime‚5204.968349018 _step‚365 & lr-AdamW/pg1‚0.0008322887202429627«Q…RËòÅ & lr-AdamW/pg2‚0.0008322887202429627 trainer/global_step‚4499 $ _timestamp‚1.788501147191166e+09 _runtime‚5204.968349018 _step‚365 & lr-AdamW/pg1‚0.0008322887202429627íœÃé`2ó- _wandbruntime‚5204  _runtime‚5204CÙBsö'ôðO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83315276 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113728670 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207384428 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27209132 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚624 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 trainer/global_step‚4499 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚370253377985688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚416671228120576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚370252452971324 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _step‚366 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113702326 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚416671252060672 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93655756 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5141044112384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5141044112384 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚365131560982960 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚377565054434816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5120909027724 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113728670 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚365131543943600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 113702326 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93655758 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93655758 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83315900 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚382705170572800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚382705170572800 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚377565054434816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5140134140928 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5140134140928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚422045196703744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚5204.969413363 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚382706098547200 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚377565036431872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚377565036431872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56106766 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5373944643072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56106768 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207358082 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5373711666688 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93655756 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚622 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 Þ&=õõO QCDeviceStatsMonitor.on_train_batch_start/requested_b×'‰=¾'ytes.all.current‚ 925014364 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93655756 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5141044112384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5141044112384 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚365131560982960 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚377565054434816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5120909027724 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113728670 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚365131543943600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 113702326 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93655758 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93655758 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83315900 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚382705170572800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚382705170572800 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚377565054434816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5140134140928 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5140134140928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚422045196703744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚5204.969413363 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚382706098547200 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚377565036431872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚377565036431872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56106766 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5373944643072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56106768 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207358082 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5373711666688 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93655756 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚622 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83315276 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113728670 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207384428 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27209132 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚624 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 trainer/global_step‚4499 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚370253377985688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚416671228120576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚370252452971324 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _step‚366 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113702326 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚416671252060672ߑö‚(:† ›¹éÔê˜t gpu.0.gpu‚42.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.916' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.229 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŠŒºTj÷‚(:ß ›¹éÔŠ”ïu disk.nvme1n1.in‚ 98.70703125 disk.sda.in‚3.546875 disk.rbd0.out‚ 1.404296875" proc.memory.rssMB‚ 3011.5546875 network.sent‚ 1.0679724e+07 disk.md0p1.out‚ 556.6171875! disk.sda2.out‚578.76513671875' proc.memory.availableMB‚ 657.6484375 disk.nvme0n1.in‚ 106.78515625# disk.nvme1n1.out‚31280.94921875 disk.md0.in‚ 4.1796875 disk.rbd2.out‚ 1.404296875 disk.md0p1.in‚ 4.1796875$ memory_percent‚95.98603248596191 network.recv‚ 1.741222e+06) disk./.usagePercent‚84.07611649685172 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sdb.in‚ 0.6328125 disk.sdb.out‚578.76513671875 disk.sda2.in‚3.546875* proc.memory.percent‚18.381071090698242 disk.sdb2.in‚ 0.6328125 disk.sda.out‚578.76513671875 disk.rbd0.in‚0 cpu‚25.1580822455338 proc.cpu.threads‚17$ disk./.usageGB‚3006.969169616699 disk.dm-0.in‚ 205.4921875 disk.dm-0.out‚ 75777.2265625 disk.md0.out‚ 556.6171875# disk.nvme0n1.out‚44496.27734375 disk.rbd2.in‚0! disk.sdb2.out‚578.76513671875 disk.sda1.in‚0 disk.sda1.out‚0VÎW¯2ø- _wandbruntime‚5206  _runtime‚5206Sm³Ãÿ&ùùM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚377646673520640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚377646655517696 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93675916 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚416761477765120 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5122026301002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚377646655517696 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5122934276214 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚422136327792128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚5206.093576026 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56118833 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚382788839106560 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5142165585920 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚422136584695296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93675918 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27214310 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 $ _timestamp‚1.788501148315668e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5374873967104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚382787911118848 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚416761453825024 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚365210578577504 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113727157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5375106930176 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83333141 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _step‚367 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207429445 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207429445 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4499 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207403073 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113753527 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5142165585920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207403073 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚370333512853718 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚382788839106560 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27214928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚416761453825024 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚365210578577504 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113727157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5375106930176 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83333141 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _step‚367 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207429445 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207429445 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4499 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207403073 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113753527 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5142165585920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207403073 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚370333512853718 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚382788839106560 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27214928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5142165585920 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚422136584695296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93675918 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27214310 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 $ _timestamp‚1.788501148315668e+09ï"¹Îå2û- _wandbruntime‚5206  _runtime‚5206Âٜݨü× /loss/FRt1/assignment_loss‚3.110465393747602 2loss/SRqqt4/assignment_loss‚1.2042047596321657 /loss/FRt4/detection_loss‚3.4120454246347602 /loss/FBt1/detection_loss‚0.7193671652496281 _step‚368 _runtime‚5206.094649551 2loss/SRqqt1/assignment_loss‚1.7710034485302262 1loss/SRqqt2/detection_loss‚2.5064830841689276 1loss/SRqqt3/detection_loss‚0.9685604555171824 /loss/FBt4/detection_loss‚2.4007746511035495 &loss/total_loss‚44.402887549438155 /loss/FRt4/assignment_loss‚3.534515619277954 2loss/SRqqt2/assignment_loss‚1.8830378153200806 /loss/FRt3/detection_loss‚1.9116482704079576 epoch‚3 % _timestamp‚1.7885011483176267e+09 0loss/FRt2/assignment_loss‚2.7362010404467583 1loss/FBt2/assignment_loss‚0.43796426234517377 1loss/FBt3/assignment_loss‚0.14968657598365098 /loss/FRt1/detection_loss‚2.0437728000347435 1loss/SRqqt1/detection_loss‚1.691103×S³dµ2175695575 0loss/FBt1/assignment_loss‚0.1722738061558361 0loss/FBt4/assignment_loss‚0.2535562859192295 /loss/SRqqt4/detection_loss‚2.87711445414103 /loss/FBt2/detection_loss‚2.2135762998035977 .loss/FBt3/detection_loss‚2.927207996447881 trainer/global_step‚4499 2loss/SRqqt3/assignment_loss‚1.0172642984741083 /loss/FRt2/detection_loss‚2.6930114590213634 0loss/FRt3/assignment_loss‚1.7680489655051912œßbýÜ 2loss/SRqqt1/assignment_loss‚1.7710034485302262 1loss/SRqqt2/detection_loss‚2.5064830841689276 1loss/SRqqt3/detection_loss‚0.9685604555171824 /loss/FBt4/detection_loss‚2.4007746511035495 &loss/total_loss‚44.402887549438155 /loss/FRt4/assignment_loss‚3.534515619277954 2loss/SRqqt2/assignment_loss‚1.8830378153200806 /loss/FRt3/detection_loss‚1.9116482704079576 epoch‚3 % _timestamp‚1.7885011483176267e+09 0loss/FRt2/assignment_loss‚2.7362010404467583 1loss/FBt2/assignment_loss‚0.43796426234517377 1loss/FBt3/assignment_loss‚0.14968657598365098 /loss/FRt1/detection_loss‚2.0437728000347435 1loss/SRqqt1/detection_loss‚1.6911032175695575 0loss/FBt1/assignment_loss‚0.1722738061558361 0loss/FBt4/assignment_loss‚0.2535562859192295 /loss/SRqqt4/detection_loss‚2.87711445414103 /loss/FBt2/detection_loss‚2.2135762998035977 .loss/FBt3/detection_loss‚2.927207996447881 trainer/global_step‚4499 2loss/SRqqt3/assignment_loss‚1.0172642984741083 /loss/FRt2/detection_loss‚2.6930114590213634 0loss/FRt3/assignment_loss‚1.7680489655051912 /loss/FRt1/assignment_loss‚3.110465393747602 2loss/SRqqt4/assignment_loss‚1.2042047596321657 /loss/FRt4/detection_loss‚3.4120454246347602 /loss/FBt1/detection_loss‚0.7193671652496281 _step‚368 _runtime‚5206.094649551ð[ñN¼’þ‚(:‡ ª¹éÔ¯ã®n gpu.0.gpu‚8.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚137.969' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.49225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö%t§nÿ‚(:ã ª¹éÔ’±p disk.rbd0.in‚0 disk.md0p1.out‚ 558.51171875! disk.sda2.out‚580.72998046875 disk.rbd2.out‚ 1.404296875 network.sent‚ 1.0776216e+07 disk.sdb1.out‚0$ memory_percent‚95.98641395568848' proc.memory.availableMB‚ 657.5859375 cpu‚25.157732742922004 disk.sda.out‚580.72998046875 disk.sdb.out‚580.72998046875 disk.sda2.in‚3.546875 disk.rbd2.in‚0 disk.sda.in‚3.546875 disk.sda1.in‚0 disk.rbd0.out‚ 1.404296875 disk.sdb1.in‚0 disk.md0p1.in‚ 4.1796875 disk.dm-0.in‚ 205.4921875 disk.nvme0n1.in‚ 106.78515625" disk.nvme0n1.out‚ 44584.3671875 disk.md0.out‚ 558.51171875 disk.dm-0.out‚75865.50390625 disk.sdb.in‚ 0.6328125) disk./.usagePercent‚84.08313657309438$ disk./.usageGB‚3007.220241546631 disk.sdb2.in‚ 0.6328125! disk.sdb2.out‚580.72998046875 disk.sda1.out‚0# disk.nvme1n1.out‚31281.13671875* proc.memory.percent‚18.381071090698242 proc.cpu.threads‚17 disk.md0.in‚ 4.1796875" proc.memory.rssMB‚ 3011.5546875 disk.nvme1n1.in‚ 98.70703125 network.recv‚ 1.746899e+06`)¶ý“€‚(:ˆ ¹¹éÔ²³­m gpu.0.gpu‚53.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.953' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.48825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù2*l‚(:á ¹¹éÔ¹‹n disk.nvme0n1.in‚ 106.78515625 disk.rbd0.out‚ 1.404296875" proc.memory.rssMB‚ 3011.5546875) disk./.usagePercent‚84.08310756142964# disk.nvme0n1.out‚45002.24609375! disk.sdb2.out‚582.21435546875 disk.sda.in‚3.546875 disk.sda.out‚582.21435546875$ memory_percent‚95.98498344421387 disk.sda1.in‚0 disk.sda2.in‚3.546875 disk.nvme1n1.in‚ 98.70703125 disk.sdb1.out‚0 disk.dm-0.out‚76283.55859375 disk.sdb.in‚ 0.6328125 proc.cpu.threads‚17 network.recv‚ 1.751385e+06 disk.rbd2.out‚ 1.404296875 disk.sdb2.in‚ 0.6328125* proc.memory.percent‚18.381071090698242 network.sent‚ 1.0874699e+07 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.md0p1.out‚ 559.94140625 disk.dm-0.in‚ 205.4921875! disk.sda2.out‚582.21435546875% disk./.usageGB‚3007.2192039489746 disk.md0.out‚ 559.94140625 disk.sdb1.in‚0 disk.md0p1.in‚ 4.1796875 disk.sdb.out‚582.21435546875' proc.memory.availableMB‚ 657.8203125 cpu‚25.157353510904603 disk.nvme1n1.out‚ 31281.3125 disk.md0.in‚ 4.1796875 disk.sda1.out‚0oÏÈ|‘‚‚(:† ȹéÔÎãºk gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚120.65' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.1625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï,x mƒ‚(:â ȹéÔÜ™ûl disk.sdb1.in‚0 disk.dm-0.out‚76429.62890625 disk.sda1.in‚0 disk.dm-0.in‚ 205.4921875 disk.sdb.in‚ 0.6328125 disk.sda.in‚3.546875 disk.rbd0.in‚0 disk.sdb.out‚583.65380859375! disk.sdb2.out‚583.65380859375 cpu‚25.15701207153182 network.sent‚ 1.087854e+07 network.recv‚ 1.755819e+06% disk./.usageGB‚3007.2192039489746 disk.rbd0.out‚ 1.404296875) disk./.usagePercent‚84.08310756142964 disk.sda.out‚583.65380859375 disk.sda2.in‚3.546875 disk.sdb2.in‚ 0.6328125# memory_percent‚95.9864854812622 proc.cpu.threads‚17 disk.rbd2.out‚ 1.404296875 disk.sdb1.out‚0 disk.md0p1.in‚ 4.1796875! disk.sda2.out‚583.65380859375" disk.nvme0n1.out‚ 45148.2109375 disk.nvme1n1.in‚ 98.70703125 disk.md0.out‚ 561.31640625 disk.rbd2.in‚0 disk.md0p1.out‚ 561.31640625 disk.nvme0n1.in‚ 106.78515625# disk.nvme1n1.out‚31281.41796875 disk.md0.in‚ 4.1796875( proc.memory.availableMB‚ 657.57421875* proc.memory.percent‚18.381071090698242 disk.sda1.out‚0" proc.memory.rssMB‚ 3011.5546875ó{2z2„- _wandbruntime‚5261  _runtime‚5261‰è÷Ç…Á & lr-AdamW/pg2‚0.0008260788152894374 trainer/global_step‚4549 % _timestamp‚1.7885012033535767e+09 _runtime‚5261.130844911 _step‚369 & lr-AdamW/pg1‚0.0008260788152894374$êœ^Ì†Æ & lr-AdamW/pg1‚0.0008260788152894374 & lr-AdamW/pg2‚0.0008260788152894374 trainer/global_step‚4549 % _timestamp‚1.7885012033535767e+09 _runtime‚5261.130844911 _step‚369ñ‡ «02‡- _wandbruntime‚5261  _runtime‚5261U‚©Ãõ'ˆïO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209637330 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚381645990723072 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚426614957322240 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5432187062784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94663758 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚374259209953632 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚369082423670800 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 209610984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚633 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5432420039168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚386843140263424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5176786282832 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 $ _timestamp‚1.788501203354937e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚381646008726016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚381645990723072 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94663756 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56710018 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56710016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27499159 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚5261.132284128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚386842212289024 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5196221565952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5196221565952 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114947228 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84209177 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚374260134967996 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 _step‚370 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚631 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚381646008726016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114973572 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 132057661443kàðú'‰ôO F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 209610984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚633 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5432420039168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚386843140263424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5176786282832 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 $ _timestamp‚1.788501203354937e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚381646008726016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚381645990723072 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94663756 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56710018 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56710016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27499159 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚5261.132284128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚386842212289024 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5196221565952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5196221565952 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114947228 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84209177 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚374260134967996 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 _step‚370 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚631 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚381646008726016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114973572 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209637330 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚381645990723072 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚426614957322240 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5432187062784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94663758 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚374259209953632 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚369082423670800 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0òuäGx2Š- _wandbruntime‚5262  _runtime‚526273ãd‹ùM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 209682347 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚386925880822784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚381727609808896 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚426706347702784 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚369161441265344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84227017 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84226402 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚374340269836026 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚381727609808896 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5198253010944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27504321 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114972059 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5433364335104 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114998429 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94683918 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114998429 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚426706103382528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5197343026176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27504934 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244320256 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94683918 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚386924952835072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220380160 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 $ _timestamp‚1.788501204448491e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚381727627811840 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚381727627811840 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚421272762987520 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚421272739047424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocat¥“U›ed‚1185 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5177903556110 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56722083 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5197343026176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94683916 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5198253010944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209682347 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56722081 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚374339344821454 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 209655975 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114972059 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 _step‚371 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚386925880822784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5433584715264 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚613 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5178811531322 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚615 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _runtime‚5262.227065709 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚4549 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209655975 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94683916 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212†pÝö'ŒþM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚386925880822784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5433584715264 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚613 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5178811531322 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚615 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _runtime‚5262.227065709 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚4549 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209655975 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94683916 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 209682347 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚386925880822784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚381727609808896 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚426706347702784 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚369161441265344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84227017 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84226402 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚374340269836026 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚381727609808896 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5198253010944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27504321 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114972059 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5433364335104 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114998429 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94683918 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114998429 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚426706103382528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5197343026176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27504934 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244320256 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94683918 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚386924952835072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220380160 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 $ _timestamp‚1.788501204448491e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚381727627811840 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚381727627811840 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚421272762987520 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚421272739047424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5177903556110 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56722083 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5197343026176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94683916 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5198253010944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209682347 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56722081 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚374339344821454 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 209655975 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114972059 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 _step‚371óŽ%-b2- _wandbruntime‚5262  _runtime‚5262TJ‚ˆ^ŽØ epoch‚3 2loss/SRqqt1/assignment_loss‚1.5397207161877304 1loss/FBt3/assignment_loss‚0.06335187577496981 0loss/FBt4/assignment_loss‚0.3355028367694623 0loss/SRqqt4/detection_loss‚2.443583739526344 _runtime‚5262.228282295 0loss/FRt2/assignment_loss‚3.7358828312472294 &loss/total_loss‚44.944053195416345 /loss/FRt1/detection_loss‚2.0493869304809778 0loss/FRt1/assignment_loss‚2.2048994451761246 0loss/FRt3/assignment_loss‚1.7822285011663275 1loss/SRqqt3/detection_loss‚1.0563060333287078 /loss/FBt2/detection_loss‚2.8367069363594055 _step‚372 2loss/SRqqt4/assignment_loss‚2.0457809648730536 .loss/FBt4/detection_loss‚2.478462578994887 trainer/global_step‚4549 % _timestamp‚1.7885012044511745e+09 /loss/FRt2/detection_loss‚2.5761118762587247 0loss/FRt4/assignment_loss‚3.7293205572211225 0loss/FBt1/assignment_loss‚0.1818306394394881 1loss/FBt2/assignment_loss‚0.14446344430325553 .loss/FRt4/detection_loss‚3.408357726491016 /loss/FBt1/detection_loss‚0.7063535531284287 2loss/SRqqt2/assignment_loss‚2.2347923359450172 2loss/SRqqt3/assignment_loss‚1.1787054940110206 /loss/FRt3/detection_loss‚1.6993148864830954 1loss/SRqqt1/detection_loss‚1.7159435749053955 1loss/SRqqt2/detection_loss‚2.2774261332610073 .loss/FBt3/detection_loss‚2.519619584083557RDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚390979254005248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95671758 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27789427 _step‚374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85102695 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95671756 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚373033320437360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 % _timestamp‚1.7885012597179706e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚390980181979648 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚385726963017216 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚431184600169984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57313266 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚378265966935940 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 211863886 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚431184838212096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚425693798565376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760‹gHû'õO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5252308990976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57313268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚390979254005248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95671758 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27789427 _step‚374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85102695 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95671756 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚373033320437360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 % _timestamp‚1.7885012597179706e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚390980181979648 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚385726963017216 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚431184600169984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57313266 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚378265966935940 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 211863886 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚431184838212096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚425693798565376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚638 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚385726945014272 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5253218962432 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85102057 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27788791 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5318.548270606 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211935249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57325333 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85119911 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5491963830272 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57325331 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211908877 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚373112338031904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5233780811218 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5492192599040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211908877 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116243331 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚391062922539008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5253430451200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95691918 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5254340435968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5254340435968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5253430451200 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211935249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚431276240808960Æ*zþ' ÿM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5234688786430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚385808564100096 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚385808564100096 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116216961 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚378347026818334 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚378346101803762 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚431276240808960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5318.548270606 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211935249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57325333 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85119911 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5491963830272 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57325331 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211908877 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚373112338031904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5233780811218 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5492192599040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211908877 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116243331 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚391062922539008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5253430451200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95691918 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5254340435968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5254340435968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5253430451200 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211935249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 Ed)Æ­À & lr-AdamW/pg1‚0.0008134728151928645 & lr-AdamW/pg2‚0.0008134728151928645 trainer/global_step‚4649 % _timestamp‚1.7885013161052651e+09 _runtime‚ 5373.88276978 _step‚377ä†Ë®Å & lr-AdamW/pg1‚0.0008134728151928645 & lr-AdamW/pg2‚0.0008134728151928645 trainer/global_step‚4649 % _timestamp‚1.7885013161052651e+09 _runtime‚ 5373.88276978 _step‚377ùªt©Ï2¯- _wandbruntime‚5373  _runtime‚5373½Ãcöö'°ðO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96679756 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214143134 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚435754804113408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚641 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚389807899305472 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96679758 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚430205107727872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5549469700608 % _timestamp‚1.7885013161064935e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5308396416000 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5308396416000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96679756 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚395117223695872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚395116295721472 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28080282 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 214116788 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5549696385536 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚643 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚378 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚395116295721472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚4649 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚389807917308416 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85996800 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚435754553488384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5288540793048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚376984183125200 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117463376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚389807917308416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5289448768052 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96679758 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚382273648932612 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117463376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57916518 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0Ó û'±õO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57916518 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96679756 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214143134 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚435754804113408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚641 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚389807899305472 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96679758 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚430205107727872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5549469700608 % _timestamp‚1.7885013161064935e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5308396416000 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5308396416000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96679756 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚395117223695872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚395116295721472 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28080282 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 214116788 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5549696385536 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚643 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚378 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚395116295721472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚4649 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚389807917308416 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85996800 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚435754553488384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5288540793048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚376984183125200 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117463376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚389807917308416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5289448768052 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96679758 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚382273648932612 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117463376úeozV2²- _wandbruntime‚5374  _runtime‚5374¯TäwG ³ùM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5309517876224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚389889536394240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117488233 \43ϸHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚435846201543168 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚430295309492224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86014024 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96699916 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚ 5374.90663367 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚382352858786070 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96699918 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5310427860992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚96699916 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5289658066326 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚382353783800642 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57928581 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 trainer/global_step‚4649 % _timestamp‚1.7885013171288712e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117461863 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚395199964255232 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214188151 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117488233 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214188151 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚377063200719744 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚395199964255232 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86014024 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96699916 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚ 5374.90663367 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚382352858786070 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96699918 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5310427860992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚96699916 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5289658066326 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚382353783800642 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57928581 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 trainer/global_step‚4649 % _timestamp‚1.7885013171288712e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117461863 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚395199964255232 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214188151 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117488233 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214188151 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚377063200719744 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚395199964255232 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚389889536394240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117488233 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚435846201543168 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚430295309492224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641ûž®L2µ- _wandbruntime‚5374  _runtime‚5374Ï"Ð$^¶Ø 2loss/SRqqt2/assignment_loss‚1.4939071720948927 epoch‚3 1loss/FBt1/assignment_loss‚0.16166694326621495 1loss/FBt3/assignment_loss‚0.07898634997984898 1loss/SRqqt2/detection_loss‚2.5293596585591636 _runtime‚5374.908553749 0loss/FRt4/assignment_loss‚2.9298160600662233 0loss/FBt2/assignment_loss‚0.0342787193819984 .loss/FRt1/detection_loss‚1.828215338708833 /loss/FRt3/detection_loss‚1.6906138485407127 1loss/SRqqt3/detection_loss‚0.9091326200172433 /loss/FRt4/detection_loss‚3.1931128025054933 1loss/SRqqt1/detection_loss‚2.2862735100694604 /loss/FBt1/detection_loss‚0.9198017905155818 trainer/global_step‚4649 0loss/FRt1/assignment_loss‚2.2691509664058684 /loss/FRt3/assignment_loss‚1.708760193165611 2loss/SRqqt3/assignment_loss‚1.0762733133056035 0loss/SRqqt4/detection_loss‚2.641106135570086 /loss/FRt2/assignment_loss‚3.240110065937042 2loss/SRqqt4/assignment_loss‚1.4604701715020032 %loss/total_loss‚41.75405755206788 /loss/FRt2/detection_loss‚2.5621461798250675 2loss/FBt4/assignment_loss‚0.020374252273662328 .loss/FBt2/detection_loss‚2.277461495664385 /loss/FBt4/detection_loss‚2.8270293633143107 _step‚380 2loss/SRqqt1/assignment_loss‚1.4696057251579053 /loss/FBt3/detection_loss‚2.1464048762406622 % _timestamp‚1.7885013171315615e+092R™Úc·Ý 1loss/FBt1/assignment_loss‚0.16166694326621495 1loss/FBt3/assignment_loss‚0.07898634997984898 1loss/SRqqt2/detection_loss‚2.5293596585591636 _runtime‚5374.908553749 0loss/FRt4/assignment_loss‚2.9298160600662233 0loss/FBt2/assignment_loss‚0.0342787193819984 .loss/FRt1/detection_loss‚1.828215338708833 /loss/FRt3/detection_loss‚1.6906138485407127 1loss/SRqqt3/detection_loss‚0.9091326200172433 /loss/FRt4/detection_loss‚3.1931128025054933 1loss/SRqqt1/detection_loss‚2.2862735100694604 /loss/FBt1/detection_loss‚0.9198017905155818 trainer/global_step‚4649 0loss/FRt1/assignment_loss‚2.2691509664058684 /loss/FRt3/assignment_loss‚1.708760193165611 2loss/SRqqt3/assignment_loss‚1.0762733133056035 0loss/SRqqt4/detection_loss‚2.641106135570086 /loss/FRt2/assignment_loss‚3.240110065937042 2loss/SRqqt4/assignment_loss‚1.4604701715020032 %loss/total_loss‚41.75405755206788 /loss/FRt2/detection_loss‚2.5621461798250675 2loss/FBt4/assignment_loss‚0.020374252273662328 .loss/FBt2/detection_loss‚2.277461495664385 /loss/FBt4/detection_loss‚2.8270293633143107 _step‚380 2loss/SRqqt1/assignment_loss‚1.4696057251579053 /loss/FBt3/detection_loss‚2.1464048762406622 % _timestamp‚1.7885013171315615e+09 2loss/SRqqt2/assignment_loss‚1.4939071720948927 epoch‚3ü¦AÆO“¸‚(:ˆ ϺéÔÃö¥l gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚120.927' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.23175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü4ž$b¹‚(:× ϺéÔÌܶm( disk./.usagePercent‚76.2242228158995 disk.md0p1.out‚ 578.671875 disk.dm-0.out‚ 79357.703125 disk.sdb.in‚ 0.6328125 disk.nvme1n1.in‚ 709.38671875 cpu‚25.153875244055236 disk.md0p1.in‚ 4.1796875 network.recv‚ 1.795923e+06 disk.sda1.out‚0 disk.sdb1.in‚0 disk.sdb.out‚ 601.591796875 disk.sda2.in‚3.546875 disk.sda2.out‚ 601.591796875# disk.nvme0n1.out‚47919.57421875 disk.sda.out‚ 601.591796875 disk.sdb2.out‚ 601.591796875$ memory_percent‚95.98734378814697 network.sent‚ 1.1147566e+07 disk.sda1.in‚0 disk.dm-0.in‚ 1610.3515625" proc.memory.rssMB‚ 3011.5546875# disk./.usageGB‚2726.14741897583 disk.md0.out‚ 578.671875* proc.memory.percent‚18.381071090698242 proc.cpu.threads‚17 disk.rbd2.out‚ 1.404296875( proc.memory.availableMB‚ 657.43359375 disk.sda.in‚3.546875 disk.md0.in‚ 4.1796875 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 1.404296875 disk.sdb1.out‚0 disk.nvme0n1.in‚ 900.96484375# disk.nvme1n1.out‚31438.12890625 disk.sdb2.in‚ 0.6328125ÕFVº‚(:„ ÞºéÔÝÁêk gpu.0.gpu‚11.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.56' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.14 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çR‰b»‚(:× ÞºéÔŽÙál network.sent‚ 1.1242473e+07 disk.md0p1.out‚ 580.39453125 disk.sdb2.in‚ 0.6328125 disk.sda1.out‚0 proc.cpu.threads‚17 disk.nvme1n1.in‚ 743.53125 disk.rbd2.out‚ 1.404296875 cpu‚25.153548448891755 disk.sda2.out‚603.3603515625 disk.md0.out‚ 580.39453125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1683.0703125 disk.sdb.out‚603.3603515625 disk.nvme0n1.out‚ 48108.09375) disk./.usagePercent‚75.78945870121262 disk.md0p1.in‚ 4.1796875 disk.nvme0n1.in‚ 939.5390625 disk.sdb2.out‚603.3603515625 disk.rbd0.out‚ 1.404296875" proc.memory.rssMB‚ 3011.5546875 disk.rbd2.in‚0 disk.sdb1.out‚0# memory_percent‚95.9888219833374% disk./.usageGB‚2710.5981483459473 disk.dm-0.out‚ 79546.390625 disk.sda.in‚3.546875 disk.sda.out‚603.3603515625 network.recv‚ 1.80188e+06 disk.sdb.in‚ 0.6328125 disk.sda2.in‚3.546875! disk.nvme1n1.out‚ 31438.296875 disk.md0.in‚ 4.1796875 disk.sdb1.in‚0( proc.memory.availableMB‚ 657.19140625* proc.memory.percent‚18.381071090698242HQô’¼‚(:‡ íºéÔ¼ø k gpu.0.gpu‚37.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.726' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.6815 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾C çk½‚(:à íºéÔ¨à¯l network.recv‚ 1.805292e+06$ disk./.usageGB‚2710.623435974121 disk.sdb2.in‚ 0.6328125 disk.sda1.in‚0# disk.nvme0n1.out‚48340.76953125 disk.md0.out‚ 582.37109375! disk.sdb2.out‚605.38232421875 disk.sdb.in‚ 0.6328125 disk.sda1.out‚0 disk.dm-0.out‚79857.02734375" proc.memory.rssMB‚ 3011.5546875! disk.sda2.out‚605.38232421875 disk.rbd2.in‚0 disk.rbd2.out‚ 1.404296875 disk.rbd0.out‚ 1.404296875 disk.md0p1.in‚ 4.1796875 disk.sdb.out‚605.38232421875 network.sent‚ 1.1247727e+07) disk./.usagePercent‚75.79016575388006 disk.md0.in‚ 4.1796875 disk.rbd0.in‚0# memory_percent‚95.9888219833374 disk.nvme0n1.in‚ 939.5390625 disk.sda.in‚3.546875 disk.sda.out‚605.38232421875 disk.sdb1.in‚0 disk.sdb1.out‚0* proc.memory.percent‚18.381071090698242 cpu‚25.15322074180171 proc.cpu.threads‚17" disk.nvme1n1.out‚ 31516.2578125 disk.md0p1.out‚ 582.37109375 disk.sda2.in‚3.546875 disk.nvme1n1.in‚ 743.53125 disk.dm-0.in‚ 1683.0703125( proc.memory.availableMB‚ 657.19140625‘ž,í2¾- _wandbruntime‚5428  _runtime‚5428©ô"¢Å¿¿ % lr-AdamW/pg1‚0.000807079347121343 % lr-AdamW/pg2‚0.000807079347121343 trainer/global_step‚4699 % _timestamp‚1.7885013707202966e+09 _runtime‚5428.497556974 _step‚381jüÚEÊÀÄ % lr-AdamW/pg2‚0.000807079347121343 trainer/global_step‚4699 % _timestamp‚1.7885013707202966e+09 _runtime‚5428.497556974 _step‚381 % lr-AdamW/pg1‚0.000807079347121343ý…+|+2Á- _wandbruntime‚5428  _runtime‚5428L‡Ä ÂðO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚399254265412096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58519768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5345326023160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5364483841024 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚399253337437696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5608449626112 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214102016 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚393888871599616 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97687756 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _runtime‚5428.498699489 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚393888853596672 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118681934 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚615 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚434716392950272 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118708278 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚399254265412096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚386280405914920 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97687756 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚434716369010176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5344418048156 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118681934 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5365393812480 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5608235524096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 216396036 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 trainer/global_step‚4699 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86889517 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚380935062852400 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚386279480900556 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚393888853596672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚399253337437696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 % _timestamp‚1.7885013707214167e+09 _step‚382 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28369136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238042112 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚440324604534272 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118708278 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216369690 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97687758 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97687758 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86888902 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58519766 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248÷5ÖYë ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5365393812480 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚440324842576384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚393888871599616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28369749 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚380935079891760 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5364483841024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944óÂF'û'ÃõO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚380935079891760 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5364483841024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚399254265412096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58519768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5345326023160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5364483841024 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚399253337437696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5608449626112 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214102016 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚393888871599616 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97687756 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _runtime‚5428.498699489 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚393888853596672 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118681934 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚615 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚434716392950272 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118708278 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚399254265412096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚386280405914920 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97687756 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚434716369010176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5344418048156 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118681934 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5365393812480 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5608235524096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 216396036 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 trainer/global_step‚4699 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86889517 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚380935062852400 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚386279480900556 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚393888853596672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚399253337437696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 % _timestamp‚1.7885013707214167e+09 _step‚382 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28369136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238042112 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚440324604534272 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118708278 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216369690 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97687758 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97687758 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86888902 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58519766 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5365393812480 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚440324842576384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚393888871599616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28369749 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784þɹ·æ2Ä- _wandbruntime‚5429  _runtime‚5429¯²š@'ÅúM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97707918 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 trainer/global_step‚4699 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97707918 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 216414681 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118733135 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚381014097486304 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5346443296646 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5366515286016 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118706765 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚386360540782950 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _step‚383 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58531833 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚440415990585344 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7885013716738808e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97707916 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58531831 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118706765 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28375517 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚393970490685440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚434806618654720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5429.451770144 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚386359615768378 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚393970472682496 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚440416247488512 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚393970472682496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5609628833792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86907350 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216441053 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚399336077983744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86906730 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 216441053 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5365605301248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚399337005971456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚399337005971456 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5345535321434 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5429.451770144 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚386359615768378 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚393970472682496 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚440416247488512 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚393970472682496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5609628833792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86907350 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216441053 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚399336077983744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86906730 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 216441053 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5365605301248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚399337005971456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚399337005971456 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5345535321434 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97707918 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 trainer/global_step‚4699 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97707918 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 216414681 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118733135 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚381014097486304 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5346443296646 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5366515286016 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118706765 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚386360540782950 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _step‚383 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58531833 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚440415990585344 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7885013716738808e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97707916 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58531831 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118706765 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28375517 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚393970490685440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚434806618654720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EbÉÜ /loss/FRt1/assignment_loss‚2.217302441596985 0loss/FRt3/assignment_loss‚2.3748246377973414 1loss/FBt1/assignment_loss‚0.13830659462965894 /loss/FRt2/detection_loss‚2.4548552320117043 .loss/FBt2/detection_loss‚2.361042891229902 _runtime‚5429.453290031 /loss/FRt4/assignment_loss‚2.874707877635956 /loss/FRt1/detection_loss‚1.8896494991957817 .loss/FBt3/detection_loss‚1.729662185907364 %loss/total_loss‚41.20169654121809 % _timestamp‚1.7885013716762981e+09 1loss/SRqqt4/assignment_loss‚1.528089992169823 /loss/FRt4/detection_loss‚2.7032834688822427 1loss/SRqqt3/detection_loss‚0.9758637415180946 /loss/FBt4/detection_loss‚2.0227953251451254 1loss/FBt4/assignment_loss‚0.29210980003222176 1loss/SRqqt4/detection_loss‚2.2442255105291093 trainer/global_step‚4699 _step‚384 1loss/FBt3/assignment_loss‚0.09667054480416866 /loss/FRt3/detection_loss‚1.6385503826079084 2loss/SRqqt1/assignment_loss‚1.0030928164099653 1loss/FBt2/assignment_loss‚0.06792877790576313 1loss/SRqqt1/detection_loss‚2.6936963087036494 1loss/SRqqt2/detection_loss‚3.2532383600870767 /loss/FRt2/assignment_loss‚2.930655587287176 2loss/SRqqt2/assignment_loss‚1.6698664943377177 /loss/FBt1/detection_loss‚0.8334251712548776 2loss/SRqqt3/assignment_loss‚1.2078528995384816 epoch‚3€Ux§dÊ‚(:Ù üºéÔ”Äßl network.sent‚ 1.1253592e+07% disk./.usageGB‚2709.6027603149414! disk.sdb2.out‚606.82177734375 disk.nvme0n1.out‚ 48429.0625 proc.cpu.threads‚17 disk.sdb2.in‚ 0.6328125 disk.sda1.in‚0 disk.sdb1.in‚0 disk.nvme1n1.in‚ 743.53125( proc.memory.availableMB‚ 657.19140625" proc.memory.rssMB‚ 3011.5546875 disk.sda.in‚3.546875 disk.md0p1.out‚ 583.74609375 network.recv‚ 1.809043e+06 disk.sda1.out‚0 disk.sdb1.out‚0 disk.md0p1.in‚ 4.1796875 disk.nvme1n1.out‚ 31518.53125 disk.rbd2.out‚ 1.404296875 disk.md0.in‚ 4.1796875# memory_percent‚95.9888219833374* proc.memory.percent‚18.381071090698242 disk.rbd2.in‚0 disk.sda.out‚606.82177734375 disk.rbd0.in‚0 disk.dm-0.in‚ 1683.0703125 disk.sdb.in‚ 0.6328125! disk.sda2.out‚606.82177734375 disk.nvme0n1.in‚ 939.5390625 disk.md0.out‚ 583.74609375 disk.rbd0.out‚ 1.404296875 disk.dm-0.out‚ 79947.59375 cpu‚25.152841679262593) disk./.usagePercent‚75.76162723526345 disk.sdb.out‚606.82177734375 disk.sda2.in‚3.546875¤xOÅË‚(:… üºéÔïŒÙl gpu.0.gpu‚53.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚178.02' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.505 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=\VÌ‚(:’ ‹»éÔɯúl gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚122.818' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚30.704500000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[xbÍ‚(:× ‹»éÔª—m disk.sdb1.in‚0 disk.sdb.out‚608.3466796875 disk.sda2.in‚3.546875 disk.nvme1n1.in‚ 743.53125# disk.nvme1n1.out‚31633.44140625 network.recv‚ 1.813798e+06 disk.sdb2.in‚ 0.6328125$ memory_percent‚95.98922729492188# proc.memory.availableMB‚657.125* proc.memory.percent‚18.381071090698242 cpu‚25.152464096007986 disk.dm-0.in‚ 1683.0703125 disk.sda2.out‚608.3466796875" disk.nvme0n1.out‚ 48430.1015625 disk.md0.in‚ 4.1796875 disk.md0.out‚ 585.2265625 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.rbd0.out‚ 1.404296875 disk.md0p1.in‚ 4.1796875 disk.rbd2.out‚ 1.404296875 disk.nvme0n1.in‚ 939.5390625" proc.memory.rssMB‚ 3011.5546875 network.sent‚ 1.136018e+07 disk.sda.out‚608.3466796875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-0.out‚80063.54296875 disk.sda.in‚3.546875 disk.sdb.in‚ 0.6328125 disk.rbd2.in‚0 disk.sdb2.out‚608.3466796875 disk.md0p1.out‚ 585.2265625) disk./.usagePercent‚75.76774080364359% disk./.usageGB‚2709.8214111328125 disk.sda1.in‚0Ã9¥ d΂(:Ù š»éÔ®×Àl disk.sda.out‚609.6943359375 disk.sda1.in‚0 disk.rbd0.out‚ 1.404296875 disk.nvme1n1.in‚ 743.53125 disk.md0.in‚ 4.1796875 disk.rbd2.out‚ 1.404296875 disk.sda.in‚3.546875 disk.rbd0.in‚0 disk.sdb2.out‚609.6943359375# memory_percent‚95.9876537322998$ disk./.usageGB‚2709.981788635254 disk.sda1.out‚0 disk.sdb.in‚ 0.6328125 disk.nvme0n1.in‚ 939.5390625 cpu‚25.152144801037483 network.sent‚ 1.1366595e+07 disk.md0p1.in‚ 4.1796875 disk.dm-0.in‚ 1683.0703125 disk.sda2.in‚3.546875 disk.md0.out‚ 586.51953125" proc.memory.rssMB‚ 3011.5546875) disk./.usagePercent‚75.77222502573505 disk.dm-0.out‚80105.94140625 disk.sda2.out‚609.6943359375 disk.sdb2.in‚ 0.6328125* proc.memory.percent‚18.381071090698242 disk.md0p1.out‚ 586.51953125# disk.nvme1n1.out‚31674.75390625 disk.sdb1.out‚0 network.recv‚ 1.818774e+06 disk.sdb1.in‚0 disk.rbd2.in‚0' proc.memory.availableMB‚ 657.3828125 disk.sdb.out‚609.6943359375 disk.nvme0n1.out‚ 48431.1875 proc.cpu.threads‚17ŒÚè “Ï‚(:ˆ š»éÔ¡óèl gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚173.059' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.26475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Àz‡Ì“Ђ(:ˆ ©»éÔ»Äæl gpu.0.gpu‚35.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚183.867' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.96675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä@¹±aÑ‚(:Ö ©»éԌЫm disk.nvme0n1.out‚ 48503.4375( proc.memory.availableMB‚ 657.12890625 disk.dm-0.in‚ 1683.0703125 network.recv‚ 1.822401e+06 disk.sda2.in‚3.546875# disk.nvme1n1.out‚31855.68359375 disk.rbd2.in‚0 disk.rbd2.out‚ 1.404296875 disk.sdb.out‚ 611.603515625 cpu‚25.151812810867725) disk./.usagePercent‚75.77613338761358 disk.sda2.out‚ 611.603515625 disk.md0.in‚ 4.1796875 disk.rbd0.out‚ 1.404296875 disk.sdb1.in‚0 network.sent‚ 1.137197e+07 disk.nvme1n1.in‚ 743.53125 disk.sdb2.out‚ 611.603515625 disk.sda.out‚ 611.603515625 disk.md0p1.in‚ 4.1796875$ disk./.usageGB‚2710.121570587158 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚95.98920345306396 proc.cpu.threads‚17 disk.nvme0n1.in‚ 939.5390625 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0p1.out‚ 588.37109375 disk.md0.out‚ 588.37109375 disk.sdb2.in‚ 0.6328125 disk.dm-0.out‚80359.12109375 disk.sdb.in‚ 0.6328125" proc.memory.rssMB‚ 3011.5546875 disk.sda.in‚3.546875* proc.memory.percent‚18.381071090698242]5ãC2Ò- _wandbruntime‚5481  _runtime‚5481¸ÏÂÇÓÁ _runtime‚5481.200889122 _step‚385 & lr-AdamW/pg1‚0.0008006273393941343 & lr-AdamW/pg2‚0.0008006273393941343 trainer/global_step‚4749 % _timestamp‚1.7885014234235258e+09‘ñ*5ÌÔÆ _step‚385 & lr-AdamW/pg1‚0.0008006273393941343 & lr-AdamW/pg2‚0.0008006273393941343 trainer/global_step‚4749 % _timestamp‚1.7885014234235258e+09 _runtime‚5481.200889122¦ô‰Y2Õ- _wandbruntime‚5481  _runtime‚5481]‹eâõ'ÖïO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚397969825890816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119926836 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚397969825890816 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 218648938 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28660061 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119953180 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚397969807887872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚444894638246400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5420571266048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚384885942579600 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5400295303264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87783077 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚439227654232576 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403390379153920 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5666739680256 trainer/global_step‚4749 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28660687 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5401203278268 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 $ _timestamp‚1.788501423424485e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚403390379153920 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚439227678172672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98695758 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59123016 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚390287162897228 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚384885959618960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87783705 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚444894393912832 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5666960073728 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98695756 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚397969807887872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5420571266048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98695756 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98695758 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚390286237882864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403390379153920 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5666739680256 trainer/global_step‚4749 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28660687 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5401203278268 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 $ _timestamp‚1.788501423424485e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚403390379153920 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚439227678172672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98695758 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59123016 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚390287162897228 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚384885959618960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87783705 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚444894393912832 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5666960073728 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98695756 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚397969807887872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5420571266048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98695756 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98695758 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚390286237882864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚397969825890816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640‚Te2Ø- _wandbruntime‚5482  _runtime‚5482ƒÜ6¨M ÙúM EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚398051426973696 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 % _timestamp‚1.7885014244966457e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚649 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5422602711040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚403473119699968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5422602711040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor• î³.on_train_batch_end/segment.all.peak‚1185 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98715918 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚398051426973696 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚398051444976640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98715918 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119978037 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚390367297765258 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚384964977213504 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87801660 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 trainer/global_step‚4749 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚439317879937024 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119978037 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28665930 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚444985780637696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119951667 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5421692726272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59135083 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5401412576542 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218693955 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚398051444976640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98715916 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚5482.274739649 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚403474047687680 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59135081 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚647 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _step‚387 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218667583 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218693955 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119951667 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87801011 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚403474047687680_öW'ÚÿM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119978037 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28665930 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚444985780637696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119951667 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5421692726272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59135083 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5401412576542 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218693955 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚398051444976640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98715916 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚5482.274739649 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚403474047687680 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59135081 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚647 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _step‚387 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218667583 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218693955 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119951667 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87801011 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚403474047687680 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚398051426973696 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 % _timestamp‚1.7885014244966457e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚649 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5422602711040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚403473119699968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5422602711040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98715918 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚398051426973696 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚398051444976640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98715918 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119978037 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚390367297765258 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚384964977213504 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87801660 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 trainer/global_step‚4749 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚439317879937024 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570ƒ kü[2Û- _wandbruntime‚5482  _runtime‚5482gVñZÜÔ 0loss/FBt2/assignment_loss‚0.4107761358403984 /loss/FRt4/detection_loss‚3.3177869103171607 0loss/SRqqt4/detection_loss‚2.487111815384456 .loss/FRt1/detection_loss‚2.043456932498367 /loss/FRt2/detection_loss‚2.1074803181303046 /loss/FRt3/detection_loss‚1.8451150537042293 epoch‚3 1loss/SRqqt2/assignment_loss‚2.074626904840653 2loss/SRqqt3/assignment_loss‚0.9581833077539311 2loss/FBt3/assignment_loss‚0.001963745619252677 /loss/FBt3/detection_loss‚3.1593281805515288 &loss/total_loss‚43.888721588533286 trainer/global_step‚4749 .loss/FRt2/assignment_loss‚3.29536671936512 0loss/SRqqt1/detection_loss‚1.873648993174235 1loss/SRqqt3/detection_loss‚1.1603222903565016 0loss/FBt1/detection_loss‚0.49349234407501563 _step‚388 /loss/SRqqt2/detection_loss‚2.76436037169053 _runtime‚ 5482.27710738 2loss/SRqqt4/assignment_loss‚1.7097099821216293 0loss/FBt1/assignment_loss‚0.1630348936268492 1loss/FBt4/assignment_loss‚0.23242634348279503 /loss/FBt2/detection_loss‚2.4608614795348225 .loss/FBt4/detection_loss‚2.959548486603631 % _timestamp‚1.7885014245000453e+09 0loss/FRt1/assignment_loss‚2.4315057799220083 0loss/FRt3/assignment_loss‚1.4897708620055248 0loss/FRt4/assignment_loss‚3.1366306326606055 2loss/SRqqt1/assignment_loss‚1.3122131052737436§«f_ÝÙ 1loss/SRqqt2/assignment_loss‚2.074626904840653 2loss/SRqqt3/assignment_loss‚0.9581833077539311 2loss/FBt3/assignment_loss‚0.001963745619252677 /loss/FBt3/detection_loss‚3.1593281805515288 &loss/total_loss‚43.888721588533286 trainer/global_step‚4749 .loss/FRt2/assignment_loss‚3.29536671936512 0loss/SRqqt1/detection_loss‚1.873648993174235 1loss/SRqqt3/detection_loss‚1.1603222903565016 0loss/FBt1/detection_loss‚0.49349234407501563 _step‚388 /loss/SRqqt2/detection_loss‚2.76436037169053 _runtime‚ 5482.27710738 2loss/SRqqt4/assignment_loss‚1.7097099821216293 0loss/FBt1/assignment_loss‚0.1630348936268492 1loss/FBt4/assignment_loss‚0.23242634348279503 /loss/FBt2/detection_loss‚2.4608614795348225 .loss/FBt4/detection_loss‚2.959548486603631 % _timestamp‚1.7885014245000453e+09 0loss/FRt1/assignment_loss‚2.4315057799220083 0loss/FRt3/assignment_loss‚1.4897708620055248 0loss/FRt4/assignment_loss‚3.1366306326606055 2loss/SRqqt1/assignment_loss‚1.3122131052737436 0loss/FBt2/assignment_loss‚0.4107761358403984 /loss/FRt4/detection_loss‚3.3177869103171607 0loss/SRqqt4/detection_loss‚2.487111815384456 .loss/FRt1/detection_loss‚2.043456932498367 /loss/FRt2/detection_loss‚2.1074803181303046 /loss/FRt3/detection_loss‚1.8451150537042293 epoch‚3„;ÀUÞ‚(:’ ¸»éÔ»ó¶m gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚119.978' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.994499999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á[Óˆgß‚(:Ü ¸»éÔ¶à›n disk.rbd0.in‚0 disk.md0p1.in‚ 4.1796875 disk.sdb.in‚ 0.6328125 disk.sda2.in‚3.546875# disk.nvme1n1.out‚31951.87890625" proc.memory.rssMB‚ 3011.5546875 disk.sda.in‚3.546875 disk.nvme1n1.in‚ 743.53125 disk.rbd2.out‚ 1.404296875 disk.sdb1.out‚0 disk.sdb.out‚613.24853515625 disk.sdb2.in‚ 0.6328125# memory_percent‚95.9885835647583 network.recv‚ 1.826713e+06% disk./.usageGB‚2710.1253395080566 disk.nvme0n1.in‚ 939.5390625 network.sent‚ 1.139075e+07 disk.sda1.out‚0 disk.sdb1.in‚0 disk.dm-0.out‚80527.33984375 disk.md0.in‚ 4.1796875( proc.memory.availableMB‚ 657.23046875) disk./.usagePercent‚75.77623876821933! disk.sda2.out‚613.24853515625" disk.nvme0n1.out‚ 48575.4609375! disk.sdb2.out‚613.24853515625 cpu‚25.15143689099872 proc.cpu.threads‚17 disk.sda.out‚613.24853515625 disk.sda1.in‚0 disk.rbd0.out‚ 1.404296875 disk.md0p1.out‚ 589.953125 disk.dm-0.in‚ 1683.0703125 disk.md0.out‚ 589.953125 disk.rbd2.in‚0* proc.memory.percent‚18.381071090698242‹;çà‚(:„ Ç»éÔ£™öl gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚176.56' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.14 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Dƒv ká‚(:à Ç»éÔêä¡m disk.sdb.out‚614.79052734375 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sdb2.in‚ 0.6328125 disk.sda1.in‚0( proc.memory.availableMB‚ 656.83984375 proc.cpu.threads‚17 disk.nvme0n1.in‚ 939.5390625" disk.nvme1n1.out‚ 32010.9609375 disk.sda.out‚614.79052734375 disk.rbd0.in‚0 disk.rbd2.out‚ 1.404296875 disk.sda.in‚3.546875 cpu‚25.151080419196877 network.recv‚ 1.831612e+06 disk.md0p1.in‚ 4.1796875 disk.md0p1.out‚ 591.43359375 disk.dm-0.in‚ 1683.0703125 disk.sdb.in‚ 0.6328125" proc.memory.rssMB‚ 3011.5546875 network.sent‚ 1.1485166e+07 disk.dm-0.out‚ 80586.859375" disk.nvme0n1.out‚ 48575.8984375 disk.nvme1n1.in‚ 743.53125 disk.md0.in‚ 4.1796875* proc.memory.percent‚18.381071090698242% disk./.usageGB‚2710.1253929138184 disk.rbd0.out‚ 1.404296875 disk.sdb1.in‚0 disk.sdb1.out‚0$ memory_percent‚95.99096775054932) disk./.usagePercent‚75.77624026146678 disk.sda2.in‚3.546875! disk.sda2.out‚614.79052734375 disk.md0.out‚ 591.43359375! disk.sdb2.out‚614.79052734375ùGD“â‚(:ˆ Ö»éÔÞŒºk gpu.0.gpu‚40.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.903' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.72575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0WÁ Žeã‚(:Ú Ö»éÔìÞÆl# disk.nvme0n1.out‚48685.19140625 disk.rbd2.in‚0 cpu‚25.150723572002224$ disk./.usageGB‚2709.848190307617 disk.sdb1.in‚0' proc.memory.availableMB‚ 656.8515625 disk.sda1.in‚0) disk./.usagePercent‚75.76848956057908 disk.nvme0n1.in‚ 939.5390625 disk.nvme1n1.in‚ 743.53125" disk.nvme1n1.out‚ 32054.5703125 disk.rbd2.out‚ 1.404296875 disk.sdb2.in‚ 0.6328125$ memory_percent‚95.99089622497559 disk.md0p1.in‚ 4.1796875 disk.md0p1.out‚ 593.46875 disk.sdb1.out‚0 disk.dm-0.out‚80739.76171875 disk.sdb.out‚616.89990234375 disk.md0.out‚ 593.46875" proc.memory.rssMB‚ 3011.5546875 disk.sda2.in‚3.546875 network.sent‚ 1.149146e+07 disk.sda.in‚3.546875 disk.sda.out‚616.89990234375 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 1.404296875 disk.md0.in‚ 4.1796875! disk.sdb2.out‚616.89990234375 network.recv‚ 1.836114e+06 disk.sdb.in‚ 0.6328125! disk.sda2.out‚616.89990234375* proc.memory.percent‚18.381071090698242 proc.cpu.threads‚17 disk.dm-0.in‚ 1683.0703125JÁrT2ä- _wandbruntime‚5534  _runtime‚5534· “(ÆåÀ $ _timestamp‚1.788501476949353e+09 _runtime‚5534.726660679 _step‚389 & lr-AdamW/pg1‚0.0007941181365999783 & lr-AdamW/pg2‚0.0007941181365999783 trainer/global_step‚47992¿> ËæÅ _runtime‚5534.726660679 _step‚389 & lr-AdamW/pg1‚0.0007941181365999783 & lr-AdamW/pg2‚0.0007941181365999783 trainer/global_step‚4799 $ _timestamp‚1.788501476949353e+09…>¶ë2ç- _wandbruntime‚5534  _runtime‚5534RòEèðO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚407528348844544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 % _timestamp‚1.7885014769504812e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121198082 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59726268 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121171738 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5476658691072 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99703756 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59726266 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚388836822306800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚407527420870144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5725267130880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121171738 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚388836839346160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5725493815808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚402050780182016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚449464206585856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99703758 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚443738939454976 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5457080533376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚402050762179072 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28950474 trainer/global_step‚4799 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99703758 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88677363 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5456172558372 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚407528348844544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚407527420870144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121198082 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚443738963395072 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ^LDeviceStatsMonito€—cŠôr.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5477568662528 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚449464457210880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚402050780182016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88676740 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚394292994865172 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _runtime‚5534.727768544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99703756 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚402050762179072 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28950474 trainer/global_step‚4799 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99703758 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88677363 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5456172558372 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚407528348844544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚407527420870144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121198082 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚443738963395072 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5477568662528 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚449464457210880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚402050780182016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88676740 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚394292994865172 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _runtime‚5534.727768544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99703756 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5476658691072 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99703756 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59726266 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚388836822306800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚407527420870144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5725267130880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121171738 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚388836839346160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0†U¹É “ê‚(:ˆ å»éÔÛ“åk gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚143.263' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.81575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t]‰Ùhë‚(:Ý å»éÔƒ¶Šm disk.rbd0.out‚ 1.404296875 disk.sdb.in‚ 0.6328125 disk.nvme0n1.in‚ 939.5390625 disk.nvme1n1.in‚ 743.53125 disk.sdb2.in‚ 0.6328125 network.recv‚ 1.839818e+06) disk./.usagePercent‚75.77621594286545 disk.md0.out‚ 594.7890625 disk.sdb1.in‚0 disk.dm-0.in‚ 1683.0703125 disk.sdb.out‚618.27783203125# disk.nvme0n1.out‚48791.30078125 disk.rbd2.out‚ 1.404296875 disk.sda1.in‚0$ memory_percent‚95.98970413208008" proc.memory.rssMB‚ 3011.5546875 network.sent‚ 1.1496752e+07 disk.sdb1.out‚0 disk.dm-0.out‚80847.12890625 disk.rbd2.in‚0! disk.nvme1n1.out‚ 32055.828125 proc.cpu.threads‚17$ disk./.usageGB‚2710.124523162842 disk.md0p1.in‚ 4.1796875! disk.sda2.out‚618.27783203125 disk.sda.in‚3.546875 cpu‚25.150356946386566 disk.md0p1.out‚ 594.7890625 disk.sda2.in‚3.546875! disk.sdb2.out‚618.27783203125 disk.sda.out‚618.27783203125& proc.memory.availableMB‚ 657.046875 disk.rbd0.in‚0 disk.md0.in‚ 4.1796875 disk.sda1.out‚0* proc.memory.percent‚18.381071090698242„§õç2ì- _wandbruntime‚5535  _runtime‚5535Y¯SØ'íúM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5726431965696 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220920485 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121196569 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88695225 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88694590 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5477780151296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚388915856940704 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5477780151296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5478690136064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220946857 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚635 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28956259 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚443829189099520 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚402132381264896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59738331 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 % _timestamp‚1.7885014781093636e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99723916 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚633 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚402132399267840 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚407610161416192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚402132399267840 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99723916 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚388915839901344 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220946857 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 _step‚391 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚407611089403904 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220920485 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121222939 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5478690136064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FP°³!îÿM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99723918 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚443829165159424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚402132381264896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59738331 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 % _timestamp‚1.7885014781093636e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99723916 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚633 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚402132399267840 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚407610161416192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚402132399267840 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99723916 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚388915839901344 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220946857 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 _step‚391 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚407611089403904 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220920485 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121222939 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5478690136064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220920485 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121196569 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88695225 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88694590 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5477780151296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚388915856940704 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5477780151296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5478690136064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220946857 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚635 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28956259 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚443829189099520 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚44 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚399753036719086 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚394231352458944 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1185 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚60641211 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚407729470800896 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 13002869760 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚407729470800896 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚407729452797952 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788501578311831e+09 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 5543087774720 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 14135738368 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1185 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 5793072156672 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189768192^òÊßÊ €ßT ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚4835 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 122529080 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚29427721 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 701042176 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 168404480 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚449834424595456 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 12553587712 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 5522365327590 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14401142784 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚449834400655360 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 701042176 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚407729452797952 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 5542404735488 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 223737080 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚570 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14401142784 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚220 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 5542404735488 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 _runtime‚5636.089682178 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 101208002 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 409073152 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 223757793 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 683039232 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 122529080 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚413271857533440 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚570 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1268 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 5543087774720 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 683039232 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13684416521 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 223737080 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚413272558575616 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 223757793 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚44 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚399753036719086 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚394231352458944 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1185 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚60641211 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚407729470800896 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 13002869760 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚407729470800896 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚407729452797952 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788501578311831e+09 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 5543087774720 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 14135738368 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1185 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 5793072156672 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189768192 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 101208000 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚399753734825894 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 698106808 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20711 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚29427503 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32006 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚60641209 JDDeviceStatsMonitor.on_valiÁ…¸k› dation_batch_start/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚413272558575616 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 681067448 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 101208002 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚90068932 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚455627496752128 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 5521684260142 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚570 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚90068712 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3725 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚218 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚394231369498304 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚615 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 101208000 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 192344576 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 5792903752192 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3725 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 122549791 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚413271857533440 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249‰ï~”^2- _wandbruntime‚5637  _runtime‚56378€d©j)‚äR TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 122537894 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1185 trainer/global_step‚4835 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 101223478 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 101223478 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 5793562007552 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚60650771 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 701042176 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 14135738368 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚44 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚455684061812224 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚449890692149248 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 5522172471026 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚455684254156800 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 409073152 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13684416521 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚615 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 168404480 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚449890668209152 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14401142784 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 101223476 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 122558605 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1185 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 122558605 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698106808 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 5543577625600 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 5542894586368 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 223782083 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 683039232 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚90083702 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 122537894 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3725 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚29432931 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 5522853538474 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 701042176 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1268 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚399804649141962 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚394281778564128 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚407783731707904 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚407783731707904 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14401142784 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 5542894586368 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚60650773 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚413327309333504 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 681067448 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚615 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚413327309333504 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189768192 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14401142784 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 _runtime‚5637.212809783 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 5793393603072 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 223761370 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20711 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚413326608291328 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚407783713704960 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚394281795603488 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1195376640 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚221 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚29433150 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚219 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚407783713704960 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13205766144Ûf\Õo)ƒéR ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 5543577625600 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 5542894586368 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 223782083 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 683039232 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚90083702 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 122537894 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3725 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚29432931 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 5522853538474 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 701042176 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1268 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚399804649141962 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚394281778564128 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚407783731707904 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚407783731707904 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14401142784 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 5542894586368 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚60650773 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚413327309333504 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 681067448 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚615 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚413327309333504 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189768192 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14401142784 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 _runtime‚5637.212809783 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 5793393603072 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 223761370 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20711 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚413326608291328 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚407783713704960 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚394281795603488 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1195376640 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚221 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚29433150 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚219 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚407783713704960 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 122537894 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1185 trainer/global_step‚4835 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 101223478 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 101223478 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 5793562007552 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚60650771 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 701042176 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 14135738368 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚44 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚455684061812224 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚449890692149248 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 5522172471026 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚455684254156800 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 409073152 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13684416521 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚615 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 168404480 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚449890668209152 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14401142784 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 101223476 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 122558605 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1185 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 122558605 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698106808Š„à1x’„‚(:‡ μéÔ®ë…m gpu.0.gpu‚42.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚185.766' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚46.4415 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fÀÊœT…‚(:É μéÔî€Üm disk.sda1.in‚0 disk.sdb.in‚ 0.6328125 disk.sda2.out‚ 629.9921875( proc.memory.availableMB‚ 623.82421875 disk.rbd2.out‚ 1.404296875 disk.sdb2.out‚ 629.9921875 disk.md0p1.out‚ 606.03125 disk.md0.out‚ 606.03125 disk.sda1.out‚0 disk.rbd0.out‚ 1.404296875 disk.sdb1.in‚0 disk.nvme0n1.in‚ 939.5390625 disk.nvme0n1.out‚ 50274.3125# disk.nvme1n1.out‚32057.32421875$ memory_percent‚96.19247913360596 disk.dm-0.out‚82331.63671875 disk.nvme1n1.in‚ 743.53125* proc.memory.percent‚18.581461906433105) disk./.usagePercent‚75.78945411480974$ disk./.usageGB‚2710.597984313965 disk.dm-0.in‚ 1683.0703125 cpu‚25.19261915182598 network.sent‚ 1.1719836e+07 disk.sda.in‚3.546875 disk.sda.out‚ 629.9921875 disk.sdb1.out‚0 disk.sdb.out‚ 629.9921875# proc.memory.rssMB‚ 3044.38671875 network.recv‚ 1.872128e+06 disk.rbd2.in‚0 disk.sdb2.in‚ 0.6328125 disk.rbd0.in‚0 disk.md0.in‚ 4.1796875 disk.md0p1.in‚ 4.1796875 disk.sda2.in‚3.546875 proc.cpu.threads‚17Ä9y¿2†- _wandbruntime‚5652  _runtime‚5652F4 ¼ ‡¿¦ ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 particle/accuracy_2_of_4‚1 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.078125 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9788074493408203 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 epoch‚3 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5641685128211975 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1266842782497406 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9917938113212585 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.47999998927116394 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_6_of_6‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6052631735801697 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09666790813207626 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21251434087753296 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.151014044880867 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4319193363189697 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9903996586799622 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4027777910232544 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5123792290687561 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.17653505504131317 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6742424368858337 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4747806787490845 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.37426382303237915 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9869073629379272 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1325691044330597 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 jet/accuracy_5_of_6‚0 *loss/val_total_loss‚42.155261376039284 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 particle/accuracy_5_of_5‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9986931085586548 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 +jet/accuracy_2_of_2‚0.10955005139112473 particle/accuracy_1_of_1‚1 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.1875 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.521006166934967 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12825995683670044 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9770114421844482 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.030631713569164276 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.552417516708374 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22145062685012817 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4897838234901428 ,jet/accuracy_3_of_3‚0.027152584865689278 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4087173342704773 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10677602142095566 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17690205574035645 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5893456339836121 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 particle/accuracy_3_of_7‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9899711608886719 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4797824025154114 9"Purity/0FRt2SRqqt1FBt/event_purity‚0.0172955971211195 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0031746032182127237 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0012210012646391988 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 +particle/specificity‚0.9530887007713318 jet/accuracy_1_of_6‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2399383932352066 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.37556472420692444 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9825268983840942 ,jet/accuracy_4_of_5‚0.018817203119397163 (particle/accuracy‚0.8531591296195984 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2577883005142212 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6666666865348816 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5299581289291382 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21058394014835358 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.01826765201985836 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07818252593278885 @&Purity/0FRt3SRqqt*FBt/eventÅÅ`&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.39542487263679504 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984036684036255 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02604166604578495 particle/accuracy_7_of_7‚0.5 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9995331764221191 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 *jet/accuracy_2_of_6‚0.5384615659713745 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37994709610939026 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9908256530761719 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9903956651687622 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.47207263112068176 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5458332896232605 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_3_of_7‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2694201171398163 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.39494046568870544 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.19481980800628662 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5384039282798767 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15019182860851288 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4150775969028473 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2242063283920288 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.43249157071113586 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.21666665375232697 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18363478779792786 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9937641620635986 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.12540462613105774 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17490023374557495 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt1SRqqt1FBt/event_purity‚0.004464285913854837 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 +jet/accuracy_1_of_1‚0.48631593585014343 *jet/accuracy_1_of_3‚0.8775138854980469 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.06382978707551956 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5202462673187256 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9629628658294678 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.36929845809936523 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 particle/accuracy_3_of_3‚1 particle/accuracy_6_of_6‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4340277910232544 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.625 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5209954977035522 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.11170551925897598 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5221264362335205 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5011778473854065 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5387915968894958 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5351190567016602 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9903167486190796 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43209877610206604 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24180321395397186 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.997383713722229 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 +jet/accuracy_2_of_3‚0.23955987393856049 particle/accuracy_3_of_5‚1 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1857907474040985 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.08825524151325226 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4333333373069763 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 *jet/accuracy_1_of_4‚0.9081180691719055 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5356603264808655 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7validation_average_jet_accuracy‚0.43816879391670227 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3448680639266968 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9801203012466431 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9894494414329529 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5199354887008667 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13701152801513672 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2619251310825348 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01881037838757038 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.031433071941137314 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9902231097221375 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.11555555462837219 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 jet/accuracy_4_of_7‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4844358265399933 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.973611056804657 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_1_of_6‚1 +"Purity/1FRt1SRqqt3FBt/event_purity‚0.25 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 trainer/global_step‚4835 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0736607164144516 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4696969985961914 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.36944442987442017 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10283096879720688 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 +jet/accuracy_4_of_6‚0.07692307978868484 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13923515379428864 7"Purity/*FRt0SRqqt1FBt/event_purity‚0.70624840259552 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9972342848777771 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19777287542819977 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.32372233271598816 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5854718089103699 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.44054052233695984 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.41684043407440186 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_5‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19593265652656555 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5195507407188416 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0008818342466838658 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5407819151878357 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4444007873535156 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5387930870056152 particle/accuracy_2_of_6‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_7‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 _runtime‚5652.966775253 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_4‚0.07431827485561371 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_7‚1 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1544165313243866 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5121832489967346 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.625 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2237173169851303 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3258351683616638 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.06578947603702545 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5333333611488342 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5452160239219666 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 _step‚395 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5577223896980286 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.14035086333751678 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1859484314918518 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.08127571642398834 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5473212599754333 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24981394410133362 *jet/accuracy_3_of_6‚0.1538461595773697 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5458085536956787 (particle/f_score‚0.39958733320236206 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4967499375343323 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5151457786560059 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3724171221256256 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1477309912443161 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5190633535385132 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_2‚1 8"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.489583283662796 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166666865348816 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 jet/accuracy_6_of_7‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_6‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.15049096941947937 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3955938220024109 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10625472664833069 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3973214328289032 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4183862507343292 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.0370548740029335 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990662932395935 8"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.689393937587738 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.20580807328224182 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9913936257362366 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9919843673706055 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,particle/sensitivity‚0.31313663721084595 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0674603134393692 5 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.99739009141922 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990662932395935 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 *jet/accuracy_1_of_5‚0.9538403153419495 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.14393939077854156 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.39455339312553406 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 jet/accuracy_2_of_7‚0 particle/accuracy_6_of_7‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6666666865348816 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5159077048301697 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5385802388191223 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 6 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.100803904235363 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41997647285461426 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,jet/accuracy_4_of_4‚0.004585538059473038 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.151014044880867 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9903250336647034 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5532407760620117 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 % _timestamp‚1.7885015951872416e+09 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0071428571827709675 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21704575419425964 particle/accuracy_2_of_2‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 *jet/accuracy_2_of_4‚0.3734537363052368 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.42008814215660095 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09278023988008499 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5554202198982239 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 jet/accuracy_1_of_7‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16540604829788208 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.4124999940395355 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0Ü»´¶ ˆĦ particle/accuracy_3_of_7‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9899711608886719 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4797824025154114 9"Purity/0FRt2SRqqt1FBt/event_purity‚0.0172955971211195 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0031746032182127237 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0012210012646391988 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 +particle/specificity‚0.9530887007713318 jet/accuracy_1_of_6‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2399383932352066 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.37556472420692444 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9825268983840942 ,jet/accuracy_4_of_5‚0.018817203119397163 (particle/accuracy‚0.8531591296195984 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2577883005142212 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6666666865348816 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5299581289291382 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21058394014835358 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.01826765201985836 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07818252593278885 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6111111044883728 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012461242265999317 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.016780618578195572 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13844852149486542 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,jet/accuracy_5_of_5‚0.008064515888690948 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_7‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/0FRt3SRfsùqqt3FBt/event_proportion‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.51531982421875 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5230837464332581 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.11973180621862411 *jet/accuracy_2_of_5‚0.4523041248321533 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5158562064170837 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.15431034564971924 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5467261672019958 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1144804134964943 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.23966580629348755 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6438000798225403 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5329033732414246 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9900000095367432 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.29855430126190186 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8020378947257996 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.43410855531692505 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3958333432674408 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3958333432674408 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5301896333694458 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5173450708389282 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2247508019208908 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5319012999534607 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7592507600784302 particle/accuracy_1_of_5‚1 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184397459030151 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.168273463845253 particle/accuracy_3_of_4‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5507737994194031 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14868181943893433 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04190574586391449 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13807319104671478 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5033292770385742 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.22959770262241364 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6315789818763733 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9833333492279053 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246032238006592 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02313748188316822 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5463753938674927 particle/accuracy_2_of_7‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9870896339416504 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20741243660449982 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5815318822860718 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.47810736298561096 particle/accuracy_1_of_4‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7"Purity/0FRt1SRqqt0FBt/event_purity‚0.51531982421875 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06831280142068863 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5674756169319153 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6527777910232544 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2564815282821655 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5159211158752441 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.006987608503550291 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.39542487263679504 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984036684036255 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02604166604578495 particle/accuracy_7_of_7‚0.5 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9995331764221191 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 *jet/accuracy_2_of_6‚0.5384615659713745 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37994709610939026 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9908256530761719 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9903956651687622 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.47207263112068176 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5458332896232605 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_3_of_7‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2694201171398163 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.39494046568870544 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.19481980800628662 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5384039282798767 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15019182860851288 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4150775969028473 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2242063283920288 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.43249157071113586 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.21666665375232697 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18363478779792786 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9937641620635986 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.12540462613105774 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17490023374557495 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt1SRqqt1FBt/event_purity‚0.004464285913854837 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 +jet/accuracy_1_of_1‚0.48631593585014343 *jet/accuracy_1_of_3‚0.8775138854980469 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.06382978707551956 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5202462673187256 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9629628658294678 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.36929845809936523 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 particle/accuracy_3_of_3‚1 particle/accuracy_6_of_6‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4340277910232544 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.625 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5209954977035522 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.11170551925897598 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5221264362335205 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5011778473854065 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5387915968894958 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5351190567016602 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9903167486190796 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43209877610206604 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24180321395397186 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.997383713722229 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 +jet/accuracy_2_of_3‚0.23955987393856049 particle/accuracy_3_of_5‚1 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1857907474040985 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.08825524151325226 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4333333373069763 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 *jet/accuracy_1_of_4‚0.9081180691719055 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5356603264808655 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7validation_average_jet_accuracy‚0.43816879391670227 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3448680639266968 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9801203012466431 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9894494414329529 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5199354887008667 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13701152801513672 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2619251310825348 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01881037838757038 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.031433071941137314 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9902231097221375 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.11555555462837219 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 jet/accuracy_4_of_7‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4844358265399933 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.973611056804657 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_1_of_6‚1 +"Purity/1FRt1SRqqt3FBt/event_purity‚0.25 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 trainer/global_step‚4835 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0736607164144516 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4696969985961914 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.36944442987442017 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10283096879720688 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 +jet/accuracy_4_of_6‚0.07692307978868484 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13923515379428864 7"Purity/*FRt0SRqqt1FBt/event_purity‚0.70624840259552 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9972342848777771 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19777287542819977 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.32372233271598816 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5854718089103699 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.44054052233695984 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.41684043407440186 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_5‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19593265652656555 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5195507407188416 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0008818342466838658 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5407819151878357 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4444007873535156 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5387930870056152 particle/accuracy_2_of_6‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_7‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 _runtime‚5652.966775253 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_4‚0.07431827485561371 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_7‚1 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1544165313243866 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5121832489967346 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.625 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2237173169851303 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3258351683616638 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.06578947603702545 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5333333611488342 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5452160239219666 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 _step‚395 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5577223896980286 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.14035086333751678 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1859484314918518 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.08127571642398834 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5473212599754333 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24981394410133362 *jet/accuracy_3_of_6‚0.1538461595773697 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5458085536956787 (particle/f_score‚0.39958733320236206 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4967499375343323 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5151457786560059 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3724171221256256 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1477309912443161 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5190633535385132 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_2‚1 8"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.489583283662796 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166666865348816 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 jet/accuracy_6_of_7‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_6‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.15049096941947937 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3955938220024109 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10625472664833069 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3973214328289032 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4183862507343292 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.0370548740029335 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990662932395935 8"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.689393937587738 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.20580807328224182 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9913936257362366 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9919843673706055 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,particle/sensitivity‚0.31313663721084595 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0674603134393692 5 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.99739009141922 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990662932395935 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 *jet/accuracy_1_of_5‚0.9538403153419495 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.14393939077854156 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.39455339312553406 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 jet/accuracy_2_of_7‚0 particle/accuracy_6_of_7‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6666666865348816 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5159077048301697 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5385802388191223 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 6 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.100803904235363 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41997647285461426 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,jet/accuracy_4_of_4‚0.004585538059473038 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.151014044880867 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9903250336647034 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5532407760620117 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 % _timestamp‚1.7885015951872416e+09 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0071428571827709675 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21704575419425964 particle/accuracy_2_of_2‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 *jet/accuracy_2_of_4‚0.3734537363052368 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.42008814215660095 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09278023988008499 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5554202198982239 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 jet/accuracy_1_of_7‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16540604829788208 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.4124999940395355 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 particle/accuracy_2_of_4‚1 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.078125 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9788074493408203 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 epoch‚3 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5641685128211975 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1266842782497406 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9917938113212585 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.47999998927116394 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_6_of_6‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6052631735801697 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09666790813207626 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21251434087753296 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.151014044880867 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4319193363189697 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9903996586799622 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4027777910232544 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5123792290687561 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.17653505504131317 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6742424368858337 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4747806787490845 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.37426382303237915 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9869073629379272 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1325691044330597 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 jet/accuracy_5_of_6‚0 *loss/val_total_loss‚42.155261376039284 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 particle/accuracy_5_of_5‚1 7 Purity/0FRt:¤\{œ *SRqqt1FBt/FBt_purity‚0.9986931085586548 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 +jet/accuracy_2_of_2‚0.10955005139112473 particle/accuracy_1_of_1‚1 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.1875 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.521006166934967 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12825995683670044 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9770114421844482 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.030631713569164276 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.552417516708374 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22145062685012817 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4897838234901428 ,jet/accuracy_3_of_3‚0.027152584865689278 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4087173342704773 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10677602142095566 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17690205574035645 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5893456339836121 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171‹=g…i‰‚(:Þ ݼéÔ¯†o disk.sdb1.out‚0 disk.sdb.in‚ 0.6328125 disk.md0.in‚ 4.1796875 disk.rbd2.in‚0 proc.cpu.threads‚17 network.sent‚ 1.1839502e+07 disk.sdb2.in‚ 0.6328125 disk.sda1.out‚0 disk.dm-0.in‚ 1683.0703125 disk.sdb.out‚631.45166015625$ memory_percent‚99.98996257781982# proc.memory.rssMB‚ 3701.21484375 disk.sda1.in‚0 disk.md0p1.in‚ 4.1796875 disk.md0p1.out‚ 607.42578125 disk.dm-0.out‚82690.34765625! disk.nvme0n1.out‚ 50632.828125 disk.nvme1n1.in‚ 743.53125 disk.sda.in‚3.546875 disk.rbd0.out‚ 1.404296875! disk.sda2.out‚631.45166015625) disk./.usagePercent‚75.78819850702561 disk.md0.out‚ 607.42578125 disk.rbd2.out‚ 1.404296875 disk.sda.out‚631.45166015625 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sda2.in‚3.546875 network.recv‚ 1.876707e+06$ disk./.usageGB‚2710.553077697754 disk.nvme0n1.in‚ 939.5390625# disk.nvme1n1.out‚32057.51953125! disk.sdb2.out‚631.45166015625) proc.memory.percent‚22.59042263031006& proc.memory.availableMB‚ 1.64453125 cpu‚25.20182472431613I<`AŠ‚(:… ݼéÔˆÚ’o gpu.0.gpu‚6.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚33.0 gpu.0.powerWatts‚86.502' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚21.6255 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»sˆÄ“‹‚(:ˆ ì¼éÔÚÕ±k gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚154.403' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.60075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£Â6í[Œ‚(:Ð ì¼éÔ™¦m disk.sda1.in‚0 disk.md0p1.out‚609.25 disk.nvme1n1.in‚ 743.53125" disk.nvme1n1.out‚ 32058.0234375 disk.rbd2.in‚0# disk./.usageGB‚2710.62540435791 disk.sdb.in‚ 0.6328125 disk.dm-0.out‚ 84240.3984375$ memory_percent‚87.95936107635498* proc.memory.percent‚18.587470054626465 proc.cpu.threads‚17 disk.rbd2.out‚ 1.404296875 disk.sdb2.in‚ 0.6328125! disk.sdb2.out‚633.33642578125 disk.md0.out‚609.25 disk.nvme0n1.out‚ 52182.375 disk.md0.in‚ 4.1796875 disk.rbd0.in‚0 disk.md0p1.in‚ 4.1796875 disk.dm-0.in‚ 1683.0703125 cpu‚25.17554257715142 disk.sdb.out‚633.33642578125 disk.sda.in‚3.546875 disk.sda.out‚633.33642578125 disk.sdb1.out‚0 network.recv‚ 1.882441e+06 disk.nvme0n1.in‚ 939.5390625 disk.sda1.out‚0 disk.rbd0.out‚ 1.404296875# proc.memory.rssMB‚ 3045.37109375 network.sent‚ 1.1951964e+07 disk.sda2.in‚3.546875 disk.sdb1.in‚0) proc.memory.availableMB‚ 1972.73828125) disk./.usagePercent‚75.79022079071463! disk.sda2.out‚633.33642578125¡U'2- _wandbruntime‚5679  _runtime‚5679jÉÿÇŽÁ & lr-AdamW/pg1‚0.0007875530952469812 & lr-AdamW/pg2‚0.0007875530952469812 trainer/global_step‚4849 % _timestamp‚1.7885016217741091e+09 _runtime‚5679.551370947 _step‚396‡ÆÀÌÆ & lr-AdamW/pg2‚0.0007875530952469812 trainer/global_step‚4849 % _timestamp‚1.7885016217741091e+09 _runtime‚5679.551370947 _step‚396 & lr-AdamW/pg1‚0.0007875530952469812Œ^*¦Á2- _wandbruntime‚5679  _runtime‚5679zaú'‘ôO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90507596 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60931922 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5563623712256 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 224686949 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚451795080566272 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5543732181554 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚451795104506368 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5815346816000 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101686746 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚601 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚457610200697344 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123000203 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5563623712256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29575674 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚415114705298432 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚415114705298432 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚395964563699952 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5542824206550 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101686746 % _timestamp‚1.7885016217751102e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122973859 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5564533683712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123000203 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚415113777324032 trainer/global_step‚4849 _runtime‚5679.552462862 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122973859 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚409550153611776 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101686744 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚401507370867142 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚415113777324032 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚409550171614720 _step‚397 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101686744 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚409550153611776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5564533683712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60931924 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚457610451322368 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5815120131072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚599 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0iÕ‡·ÿ'’ùO O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚409550171614720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90508197 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 224660603 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5563623712256 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 224686949 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚451795080566272 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5543732181554 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚451795104506368 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5815346816000 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101686746 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚601 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚457610200697344 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123000203 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5563623712256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29575674 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚415114705298432 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚415114705298432 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚395964563699952 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5542824206550 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101686746 % _timestamp‚1.7885016217751102e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122973859 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5564533683712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123000203 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚415113777324032 trainer/global_step‚4849 _runtime‚5679.552462862 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122973859 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚409550153611776 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101686744 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚401507370867142 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚415113777324032 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚409550171614720 _step‚397 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101686744 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚409550153611776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5564533683712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60931924 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚457610451322368 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5815120131072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚599 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90507596 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60931922 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚451885306270720 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5543941479828 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚409631790700544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5565655157248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90525347 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60943989 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29581992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚451885330210816 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101706906 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122998690 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚457701595302912 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5816289032192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚396043564255136 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122998690 WCDhì=eviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚415197445857792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚415196517870080 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚409631790700544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚409631772697600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101706904 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 224705594 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 % _timestamp‚1.7885016228944564e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123025060 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚401587505734964 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224731966 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29581360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚457701850108928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚90525981 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚401588430749536 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60943987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _runtime‚5680.673230467 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚415197445857792 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5565655157248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90525347 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60943989 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29581992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚451885330210816 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101706906 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122998690 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚457701595302912 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5816289032192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚396043564255136 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122998690 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚415197445857792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚415196517870080 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚409631790700544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚409631772697600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101706904 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 224705594 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 % _timestamp‚1.7885016228944564e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123025060 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚401587505734964 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224731966 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29581360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚457701850108928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚90525981 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚401588430749536 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60943987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _runtime‚5680.673230467 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚415197445857792 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚451885306270720 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5543941479828 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0Žy2–- _wandbruntime‚5680  _runtime‚56805¤<ªZ—Ô 1loss/FBt3/assignment_loss‚0.09245060284393376 1loss/SRqqt1/detection_loss‚1.9700539857149124 1loss/SRqqt3/detection_loss‚0.9556925440828006 2loss/SRqqt1/assignment_loss‚1.6804959151893855 2loss/SRqqt4/assignment_loss‚1.5111472478934698 /loss/FRt1/detection_loss‚1.8826932024950471 /loss/FRt4/detection_loss‚3.0927826315164566 1loss/SRqqt4/detection_loss‚2.6501966608422145 %loss/total_loss‚43.78291956317697 _runtime‚ 5680.67615242 2loss/SRqqt3/assignment_loss‚1.1905253215471194 /loss/FBt1/assignment_loss‚0.187818512388445 .loss/FRt3/detection_loss‚1.475795782238677 /loss/FBt4/detection_loss‚3.2863960385322573 trainer/global_step‚4849 0loss/FBt4/assignment_loss‚0.3935153116724905 .loss/FRt2/detection_loss‚2.194000152625003 0loss/SRqqt2/detection_loss‚2.777591198682785 0loss/FRt3/assignment_loss‚1.5966260381366895 1loss/FBt2/assignment_loss‚0.07010104551833744 /loss/FBt3/detection_loss‚1.7723920101469213 /loss/FRt1/assignment_loss‚2.415853456009266 /loss/FBt1/detection_loss‚0.6808775777633234 /loss/FRt2/assignment_loss‚3.569439669824996 0loss/FRt4/assignment_loss‚3.1449691504240036 % _timestamp‚1.7885016228980234e+09 2loss/SRqqt2/assignment_loss‚1.5751543893980293 /loss/FBt2/detection_loss‚3.6163511176904044 epoch‚4 _step‚399nq ò_˜Ù /loss/FRt1/assignment_loss‚2.415853456009266 /loss/FBt1/detection_loss‚0.6808775777633234 /loss/FRt2/assignment_loss‚3.569439669824996 0loss/FRt4/assignment_loss‚3.1449691504240036 % _timestamp‚1.7885016228980234e+09 2loss/SRqqt2/assignment_loss‚1.5751543893980293 /loss/FBt2/detection_loss‚3.6163511176904044 epoch‚4 _step‚399 1loss/FBt3/assignment_loss‚0.09245060284393376 1loss/SRqqt1/detection_loss‚1.9700539857149124 1loss/SRqqt3/detection_loss‚0.9556925440828006 2loss/SRqqt1/assignment_loss‚1.6804959151893855 2loss/SRqqt4/assignment_loss‚1.5111472478934698 /loss/FRt1/detection_loss‚1.8826932024950471 /loss/FRt4/detection_loss‚3.0927826315164566 1loss/SRqqt4/detection_loss‚2.6501966608422145 %loss/total_loss‚43.78291956317697 _runtime‚ 5680.67615242 2loss/SRqqt3/assignment_loss‚1.1905253215471194 /loss/FBt1/assignment_loss‚0.187818512388445 .loss/FRt3/detection_loss‚1.475795782238677 /loss/FBt4/detection_loss‚3.2863960385322573 trainer/global_step‚4849 0loss/FBt4/assignment_loss‚0.3935153116724905 .loss/FRt2/detection_loss‚2.194000152625003 0loss/SRqqt2/detection_loss‚2.777591198682785 0loss/FRt3/assignment_loss‚1.5966260381366895 1loss/FBt2/assignment_loss‚0.07010104551833744 /loss/FBt3/detection_loss‚1.7723920101469213ßä“™‚(:ˆ û¼éÔÖ·Çm gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚180.473' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.11825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µE²{iš‚(:Þ û¼éÔ¿‡Ún disk.sdb2.in‚ 0.6328125 disk.sdb1.out‚0 cpu‚25.175200115881466 disk.md0.out‚ 611.02734375 disk.dm-0.in‚ 1683.0703125 disk.sdb1.in‚0 network.sent‚ 1.196845e+07 disk.sdb.in‚ 0.6328125 disk.rbd2.in‚0 disk.md0p1.out‚ 611.02734375 network.recv‚ 1.886554e+06 disk.sda1.out‚0 disk.dm-0.out‚ 84478.2109375( disk./.usagePercent‚75.7937244825343% disk./.usageGB‚2710.7507133483887 disk.sda2.out‚635.1669921875# disk.nvme0n1.out‚52420.09765625 disk.sda.in‚3.546875# memory_percent‚87.9584550857544# proc.memory.rssMB‚ 3045.37109375* proc.memory.percent‚18.587470054626465 disk.nvme1n1.in‚ 743.53125# disk.nvme1n1.out‚32058.11328125 disk.sdb.out‚635.1669921875 disk.nvme0n1.in‚ 939.5390625 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda2.in‚3.546875 disk.md0.in‚ 4.1796875 disk.sdb2.out‚635.1669921875 disk.sda.out‚635.1669921875 disk.sda1.in‚0 disk.rbd0.out‚ 1.404296875) proc.memory.availableMB‚ 1972.88671875 disk.md0p1.in‚ 4.1796875 disk.rbd2.out‚ 1.4042968754]k¶‘›‚(:† Š½éÔö¥›m gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.028' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.007 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ·¹Tnœ‚(:ã Š½éÔŒî„n disk.sdb2.in‚ 0.6328125 disk.sda1.out‚0! disk.sda2.out‚636.60107421875 cpu‚25.174834383149513) disk./.usagePercent‚75.79720812217391% disk./.usageGB‚2710.8753051757812 disk.sdb1.out‚0 disk.md0p1.in‚ 4.1796875 disk.md0p1.out‚ 612.3984375 network.sent‚ 1.2063394e+07" disk.nvme1n1.out‚ 32058.3203125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 1.404296875 disk.sdb1.in‚0 disk.dm-0.out‚84607.83984375 disk.nvme0n1.in‚ 939.5390625# disk.nvme0n1.out‚52549.51953125 disk.nvme1n1.in‚ 743.53125 disk.dm-0.in‚ 1683.0703125 disk.sdb.out‚636.60107421875 disk.md0.in‚ 4.1796875 disk.rbd2.in‚0) proc.memory.availableMB‚ 1972.64453125 disk.sda2.in‚3.546875 disk.sda.in‚3.546875# proc.memory.rssMB‚ 3045.43359375 proc.cpu.threads‚17 disk.rbd2.out‚ 1.404296875! disk.sdb2.out‚636.60107421875 disk.sda.out‚636.60107421875 disk.sdb.in‚ 0.6328125$ memory_percent‚87.95993328094482* proc.memory.percent‚18.587851524353027 network.recv‚ 1.892356e+06 disk.md0.out‚ 612.3984375øÍ%’‚(:‡ ™½éÔ£Ž‰m gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.942' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4855 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–è4 \ž‚(:Ñ ™½éÔå´çm disk.sda.out‚ 637.958984375 disk.rbd0.out‚ 1.404296875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.out‚84973.37109375 disk.sdb.out‚ 637.958984375 disk.sda1.in‚0 disk.dm-0.in‚ 1683.0703125 disk.sda2.out‚ 637.958984375 proc.cpu.threads‚17 network.recv‚ 1.895435e+06 disk.md0p1.in‚ 4.1796875) disk./.usagePercent‚75.80419641357801% disk./.usageGB‚2711.1252403259277 disk.rbd2.out‚ 1.404296875 disk.rbd0.in‚0 cpu‚25.174466636481906 disk.nvme0n1.out‚ 52914.9375 disk.md0.in‚ 4.1796875 disk.rbd2.in‚0 disk.sda2.in‚3.546875$ memory_percent‚87.95933723449707* proc.memory.percent‚18.587851524353027 network.sent‚ 1.2068059e+07# disk.nvme1n1.out‚32058.43359375 disk.md0.out‚613.6875 disk.sda1.out‚0 disk.nvme1n1.in‚ 743.53125 disk.sdb.in‚ 0.6328125( proc.memory.availableMB‚ 1972.7421875# proc.memory.rssMB‚ 3045.43359375 disk.nvme0n1.in‚ 939.5390625 disk.sda.in‚3.546875 disk.md0p1.out‚613.6875 disk.sdb2.in‚ 0.6328125 disk.sdb2.out‚ 637.958984375ô&w“Ÿ‚(:ˆ ¨½éÔ÷šm gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚131.173' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.79325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Lùî1m ‚(:â ¨½éÔàÙßm proc.cpu.threads‚17 disk.md0p1.in‚ 4.1796875 disk.dm-0.in‚ 1683.0703125! disk.nvme0n1.out‚ 53077.203125* proc.memory.percent‚18.587851524353027! disk.sdb2.out‚639.79833984375 disk.rbd0.out‚ 1.404296875 disk.sdb1.in‚0 disk.md0p1.out‚ 615.49609375! disk.sda2.out‚639.79833984375 disk.rbd2.in‚0 disk.rbd2.out‚ 1.404296875 network.recv‚ 1.899831e+06 disk.sda1.out‚0" disk.nvme1n1.out‚ 32058.6328125 disk.md0.in‚ 4.1796875 disk.sda.out‚639.79833984375) proc.memory.availableMB‚ 1972.75390625) disk./.usagePercent‚75.80419641357801$ memory_percent‚87.95926570892334# proc.memory.rssMB‚ 3045.43359375 network.sent‚ 1.2074064e+07% disk./.usageGB‚2711.1252403259277 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sdb.out‚639.79833984375 disk.nvme0n1.in‚ 939.5390625 disk.md0.out‚ 615.49609375 disk.sda1.in‚0 disk.nvme1n1.in‚ 743.53125 disk.sdb2.in‚ 0.6328125 disk.sda2.in‚3.546875 disk.sda.in‚3.546875 cpu‚25.174099393708765 disk.dm-0.out‚ 85121.0859375 disk.sdb.in‚ 0.6328125£Îp¢2¡- _wandbruntime‚5737  _runtime‚5737žÔeÆÅ¢¿ & lr-AdamW/pg1‚0.0007809335834799209 & lr-AdamW/pg2‚0.0007809335834799209 trainer/global_step‚4899 $ _timestamp‚1.788501679377532e+09 _runtime‚ 5737.15492472 _step‚400ÆÖ=–检 & lr-AdamW/pg1‚0.0007809335834799209 & lr-AdamW/pg2‚0.0007809335834799209 trainer/global_step‚4899 $ _timestamp‚1.788501679377532e+09 _runtime‚ 5737.15492472 _step‚400?WÛ}2¤- _wandbruntime‚5737  _runtime‚5737z Ÿ|®$¥ôO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5873361284096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5873590066176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚405515052863814 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚405514127849450 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚413631107902976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124218761 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚419251747014656 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚413631107902976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102694744 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚419250819040256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚419251747014656 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5619711137280 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_baêÛ•5Ltch_start/active.large_pool.freed‚ 102694744 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61535172 _step‚401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚462179979794944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5598701461658 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚456306389728768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚419250819040256W;ºÿ'¦ùO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚413631125905920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226939851 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102694746 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124245105 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚419250819040256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚419251747014656 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5619711137280 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102694744 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61535172 _step‚401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚462179979794944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5598701461658 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚456306389728768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚419250819040256 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5873361284096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5873590066176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚405515052863814 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚405514127849450 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚413631107902976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124218761 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚419251747014656 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚413631107902976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102694744 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚462271374440448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚399994461021696 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚413712744991744 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 226958496 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226984868 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5874530238464 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124269962 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61547239 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚5738.201755398 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91414778 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5874759007232 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102714906 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29868162 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124243592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚462271121731584 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚456396615433216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226984868 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚456396591493120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885016804237058e+09 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5600726710148 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102714906 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5621742582272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚419333559586304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61547237 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 226958496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5621742582272 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚402 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚462271374440448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚399994461021696 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚413712744991744 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 226958496 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226984868 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5874530238464 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124269962 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61547239 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚5738.201755398 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91414778 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5874759007232 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102714906 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29868162 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124243592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚462271121731584 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚456396615433216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226984868 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚456396591493120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885016804237058e+09 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5600726710148 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102714906 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5621742582272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚419333559586304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61547237 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 226958496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0’y¹.¥2ª- _wandbruntime‚5738  _runtime‚5738š÷£’]«× 1loss/SRqqt2/assignment_loss‚1.190521872885849 2loss/SRqqt3/assignment_loss‚1.0051444350757588 %loss/total_loss‚41.55797420108798 0loss/FBt2/assignment_loss‚0.2431405273046039 /loss/FRt4/detection_loss‚3.0485226488113404 /loss/FRt1/detection_loss‚1.9141201307687625 1loss/SRqqt1/detection_loss‚2.1399022097940796 1loss/SRqqt3/detection_loss‚1.0358805154466022 1loss/SRqqt4/detection_loss‚2.5717960028879103 /loss/FBt3/detection_loss‚2.5715749661127725 0loss/FRt2/assignment_loss‚3.2348963572428775 1loss/FBt3/assignment_loss‚0.11401964492524207 1loss/FBt4/assignment_loss‚0.37021713191175615 .loss/FRt2/detection_loss‚2.452365679236559 /loss/FBt1/detection_loss‚0.8444359080315056 % _timestamp‚1.7885016804263623e+09 /loss/FRt3/assignment_loss‚1.502293001202976 2loss/SRqqt1/assignment_loss‚1.5730680500467618 /loss/FRt3/detection_loss‚1.6630151258671986 trainer/global_step‚4899 _runtime‚5738.203352915 /loss/FRt4/assignment_loss‚2.632927372455597 2loss/SRqqt4/assignment_loss‚1.7617513212946154 epoch‚4 0loss/FRt1/assignment_loss‚2.4417693462482717 /loss/FBt4/detection_loss‚2.3891466557979584 _step‚403 0loss/FBt1/assignment_loss‚0.1072360385149457 1loss/SRqqt2/detection_loss‚2.7069297406984414 /loss/FBt2/detection_loss‚2.0432995185256004ñ6íɯ¬Ü 0loss/FBt2/assignment_loss‚0.2431405273046039 /loss/FRt4/detection_loss‚3.0485226488113404 /loss/FRt1/detection_loss‚1.9141201307687625 1loss/SRqqt1/detec´®E³tion_loss‚2.1399022097940796 1loss/SRqqt3/detection_loss‚1.0358805154466022 1loss/SRqqt4/detection_loss‚2.5717960028879103 /loss/FBt3/detection_loss‚2.5715749661127725 0loss/FRt2/assignment_loss‚3.2348963572428775 1loss/FBt3/assignment_loss‚0.11401964492524207 1loss/FBt4/assignment_loss‚0.37021713191175615 .loss/FRt2/detection_loss‚2.452365679236559 /loss/FBt1/detection_loss‚0.8444359080315056 % _timestamp‚1.7885016804263623e+09 /loss/FRt3/assignment_loss‚1.502293001202976 2loss/SRqqt1/assignment_loss‚1.5730680500467618 /loss/FRt3/detection_loss‚1.6630151258671986 trainer/global_step‚4899 _runtime‚5738.203352915 /loss/FRt4/assignment_loss‚2.632927372455597 2loss/SRqqt4/assignment_loss‚1.7617513212946154 epoch‚4 0loss/FRt1/assignment_loss‚2.4417693462482717 /loss/FBt4/detection_loss‚2.3891466557979584 _step‚403 0loss/FBt1/assignment_loss‚0.1072360385149457 1loss/SRqqt2/detection_loss‚2.7069297406984414 /loss/FBt2/detection_loss‚2.0432995185256004 1loss/SRqqt2/assignment_loss‚1.190521872885849 2loss/SRqqt3/assignment_loss‚1.0051444350757588 %loss/total_loss‚41.55797420108798“×–=º­‚(:’ ·½éÔ¥ÙÒl gpu.0.gpu‚44.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚118.474' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.618500000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XdZ&d®‚(:Ù ·½éÔ¬¡‚n disk.dm-0.out‚85260.5 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0n1.out‚ 53201.7109375$ memory_percent‚87.96088695526123 disk.rbd0.in‚0 disk.md0.in‚ 4.1796875$ disk./.usageGB‚2711.125217437744 disk.sdb.out‚641.46923828125 disk.sda2.in‚3.546875! disk.sda2.out‚641.46923828125 disk.nvme0n1.in‚ 939.5390625 disk.sdb2.in‚ 0.6328125* proc.memory.percent‚18.587851524353027 cpu‚25.17368765906917 disk.md0p1.out‚ 617.1015625 disk.rbd2.out‚ 1.404296875 disk.sda.in‚3.546875 network.recv‚ 1.910766e+06 disk.rbd0.out‚ 1.404296875 disk.sdb1.in‚0 disk.md0p1.in‚ 4.1796875" disk.nvme1n1.out‚ 32058.7890625 disk.md0.out‚ 617.1015625) proc.memory.availableMB‚ 1972.48828125# proc.memory.rssMB‚ 3045.43359375 disk.sdb1.out‚0 disk.dm-0.in‚ 1683.0703125 disk.nvme1n1.in‚ 743.53125! disk.sdb2.out‚641.46923828125 disk.sda.out‚641.46923828125 network.sent‚ 1.2096137e+07 disk.rbd2.in‚0) disk./.usagePercent‚75.80419577361481 disk.sdb.in‚ 0.6328125 proc.cpu.threads‚17’â°ê“¯‚(:ˆ ƽéÔ·»o gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚117.525' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.38125 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 7äxi°‚(:Þ ƽéÔÜ”Ñq disk.sda.out‚642.7216796875* proc.memory.percent‚18.587851524353027 cpu‚25.173247596264023 disk.sdb.in‚ 0.6328125 disk.sda2.in‚3.546875 disk.sdb1.out‚0 disk.md0p1.in‚ 4.1796875 disk.nvme0n1.in‚ 939.5390625# disk.nvme1n1.out‚32058.98828125( disk./.usagePercent‚75.7882566370156 disk.sdb.out‚642.7216796875 disk.nvme1n1.in‚ 743.53125 disk.rbd2.out‚ 1.404296875 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-0.out‚85652.04296875 proc.cpu.threads‚17 network.sent‚ 1.2191279e+07 disk.rbd2.in‚0 disk.rbd0.out‚ 1.404296875# proc.memory.rssMB‚ 3045.43359375% disk./.usageGB‚2710.5551567077637 disk.sda2.out‚642.7216796875" disk.nvme0n1.out‚ 53593.0546875$ memory_percent‚87.96231746673584 disk.md0.out‚ 618.2890625 disk.sdb1.in‚0 disk.md0p1.out‚ 618.2890625 disk.dm-0.in‚ 1683.0703125 disk.sdb2.in‚ 0.6328125 disk.sdb2.out‚642.7216796875 disk.sda.in‚3.546875 disk.sda1.out‚0) proc.memory.availableMB‚ 1972.25390625 network.recv‚ 1.916806e+06 disk.md0.in‚ 4.1796875ª’oP‘±‚(:† Õ½éÔÑ–Íl gpu.0.gpu‚29.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚121.876' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.469 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïù¾dd²‚(:Ù Õ½éÔ°´m disk.nvme1n1.in‚ 743.53125# disk.nvme1n1.out‚32059.10546875 disk.md0.out‚ 620.21875 disk.sdb2.in‚ 0.6328125 disk.sda1.out‚0 disk.rbd0.out‚ 1.404296875* proc.memory.percent‚18.587851524353027 proc.cpu.threads‚17) disk./.usagePercent‚75.77579623367156 disk.sda1.in‚0! disk.sda2.out‚644.71337890625 disk.nvme0n1.in‚ 939.5390625 disk.sdb1.in‚0 disk.sdb.in‚ 0.6328125 disk.sda2.in‚3.546875 disk.rbd2.out‚ 1.404296875# proc.memory.rssMB‚ 3045.43359375 disk.md0.in‚ 4.1796875 cpu‚25.17284060933253 network.recv‚ 1.920187e+06 disk.sda.out‚644.71337890625 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.dm-0.out‚ 85905.375 disk.sdb.out‚644.71337890625 disk.rbd2.in‚0 network.sent‚ 1.2198737e+07! disk.sdb2.out‚644.71337890625 disk.dm-0.in‚ 1683.0703125$ memory_percent‚91.97564125061035( proc.memory.availableMB‚ 1314.7109375% disk./.usageGB‚2710.1095123291016 disk.sda.in‚3.546875 disk.md0p1.in‚ 4.1796875 disk.md0p1.out‚ 620.21875# disk.nvme0n1.out‚53846.269531258ŹN‘³‚(:† ä½éÔÕÂl gpu.0.gpu‚55.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚150.724' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.681 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_ãÚd´‚(:Ù ä½éÔ­Šül$ disk./.usageGB‚2710.135383605957 disk.sda2.in‚3.546875 disk.sda2.out‚646.1337890625 disk.nvme0n1.in‚ 939.5390625 disk.md0p1.in‚ 4.1796875' proc.memory.availableMB‚ 1314.828125 proc.cpu.threads‚17 network.sent‚ 1.2204872e+07 disk.dm-0.out‚ 85906.28125 disk.sdb.in‚ 0.6328125 disk.md0.in‚ 4.1796875 disk.sdb2.in‚ 0.6328125 disk.sda1.in‚0 disk.sdb1.out‚0# disk.nvme0n1.out‚53846.91015625 disk.rbd0.in‚0* proc.memory.percent‚18.587851524353027 cpu‚25.172482138062712 disk.nvme1n1.in‚ 743.53125 disk.sdb1.in‚0 disk.md0p1.out‚ 621.6015625 network.recv‚ 1.92497e+06# disk.nvme1n1.out‚32059.37109375 disk.md0.out‚ 621.6015625 disk.rbd0.out‚ 1.404296875 disk.dm-0.in‚ 1683.0703125 disk.sdb2.out‚646.1337890625 disk.sda.out‚646.1337890625# proc.memory.rssMB‚ 3045.43359375 disk.sdb.out‚646.1337890625 disk.rbd2.in‚0 disk.rbd2.out‚ 1.404296875 disk.sda.in‚3.546875) disk./.usagePercent‚75.77651960540041 disk.sda1.out‚0$ memory_percent‚91.97492599487305 XEg2µ- _wandbruntime‚5794  _runtime‚5794O %¶Å¶¿ % lr-AdamW/pg2‚0.000774260980795128 trainer/global_step‚4949 % _timestamp‚1.7885017362474723e+09 _runtime‚5794.024825684 _step‚404 % lr-AdamW/pg1‚0.000774260980795128˜opœÊ·Ä % _timestamp‚1.7885017362474723e+09 _runtime‚5794.024825684 _step‚404 % lr-AdamW/pg1‚0.000774260980795128 % lr-AdamW/pg2‚0.000774260980795128 trainer/global_step‚4949”"¬ 2¸- _wandbruntime‚5794  _runtime‚5794ä»Wú'¹ôO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92289890 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5676708533760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92289276 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚417712080197120 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚417712062194176 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚614 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62138424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚423388788730880 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103702744 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 229192753 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30151466 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚423388788730880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 _runtime‚5794.026216541 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103702744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚460817651011072 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5931766567936 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚409520884831758 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 % _timestamp‚1.7885017362487192e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚409521809846122 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5676708533760 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125490007 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚466749417579008 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚417712062194176 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125463663 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5655486691770 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125490007 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 229166407 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125463663 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _step‚405 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚423387860756480 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103702746 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚4949 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5931999544320 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚403866323154352 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚423387860756480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0Äœõÿ'ºùO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚460817674951168 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5654578716766 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚409520884831758 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 % _timestamp‚1.7885017362487192e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚409521809846122 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5676708533760 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125490007 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚466749417579008 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚417712062194176 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125463663 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5655486691770 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125490007 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 229166407 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125463663 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _step‚405 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚423387860756480 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103702746 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚4949 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5931999544320 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚403866323154352 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚423387860756480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92289890 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5676708533760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92289276 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚417712080197120 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚417712062194176 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚614 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62138424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚423388788730880 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103702744 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 229192753 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30151466 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚423388788730880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 _runtime‚5794.026216541 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103702744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚460817651011072 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5931766567936 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92307034 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚460907900655616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 229237770 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92307651 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/a4oòê3llocation.all.allocated‚ 229237770 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚417793699282944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125514864 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 125488494 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚466840804467712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 % _timestamp‚1.7885017373085558e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚423471529290240 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚615 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5676920022528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5677830007296 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚417793681280000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5677830007296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚460907876715520 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚403945323709536 _step‚406 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5676920022528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚423470601302528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5655695990044 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125514864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚409601944714152 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚617 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30157162 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚466841061370880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 229237770 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚417793699282944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125514864 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 125488494 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚466840804467712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 % _timestamp‚1.7885017373085558e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚423471529290240 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚615 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5676920022528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5677830007296 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚417793681280000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5677830007296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚460907876715520 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚403945323709536 _step‚406 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5676920022528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚423470601302528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5655695990044 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125514864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚409601944714152 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚617 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30157162 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚466841061370880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92307034 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚460907900655616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 229237770 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92307651 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚407817185842192 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30439685 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚413528566828430 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4999 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104710746 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚471318989445632 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126708565 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚427524902472704 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104710744 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚427524902472704 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62741672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259013632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126734909 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126708565 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚427525830447104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 231445655 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30440291 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5990053212160 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126708565 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚427524902472704 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104710744 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚427524902472704 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62741672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259013632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126734909 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126708565 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚427525830447104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 231445655 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30440291 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5990053212160 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚407817185842192 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30439685 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚413528566828430 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4999™tQ†_2Ï- _wandbruntime‚5852  _runtime‚5852 'ÐþM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93199077 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚427608571006464 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 231464300 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚427607643018752 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126759766 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104730904 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93199675 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚421874635571200 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5733007447552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚471410640409600 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126733396 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5733917432320 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5711573245152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚421874653574144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4999 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231490672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 _step‚410 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126759766 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30445340 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚465419185878016 % _timestamp‚1.7885017948839986e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30445936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5733007447552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚598 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚596׌ô7&уN TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126759766 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30445340 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚465419185878016 % _timestamp‚1.7885017948839986e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30445936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5733007447552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚598 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚596 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93199077 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚427608571006464 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 231464300 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚427607643018752 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126759766 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104730904 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93199675 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚421874635571200 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5733007447552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚471410640409600 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126733396 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5733917432320 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5711573245152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚421874653574144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4999 L>DeviceStatsMonitor.on_train_batch <Ý]Ò_end/allocation.all.allocated‚ 231490672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 _step‚410 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0š‹yu¹2Ò- _wandbruntime‚5852  _runtime‚5852éÃzZÓÔ 1loss/FBt1/assignment_loss‚0.15569868052657515 /loss/FBt1/detection_loss‚0.7168657053403105 2loss/SRqqt1/assignment_loss‚1.5066013510028522 0loss/SRqqt1/detection_loss‚2.251903818299373 /loss/FBt3/detection_loss‚2.4520561893781028 0loss/FRt3/assignment_loss‚1.3051593426395864 /loss/FRt4/assignment_loss‚4.494871122496469 1loss/FBt2/assignment_loss‚0.17267373819558998 % _timestamp‚1.7885017948866193e+09 2loss/SRqqt4/assignment_loss‚1.7664857943143164 1loss/SRqqt3/detection_loss‚0.9972016336815851 _runtime‚5852.666835374 _step‚411 /loss/FRt2/detection_loss‚2.4412405961193144 2loss/SRqqt3/assignment_loss‚0.9810435458312969 0loss/FBt4/assignment_loss‚0.1117704096542928 /loss/FRt3/detection_loss‚1.4434261380092186 1loss/SRqqt2/detection_loss‚2.5790408754348757 0loss/SRqqt4/detection_loss‚2.744063542357513 .loss/FBt2/detection_loss‚2.412618786096573 /loss/FRt1/assignment_loss‚2.743676861578768 3loss/FBt3/assignment_loss‚0.0014193959240553038 -loss/FRt1/detection_loss‚1.81983613119122 .loss/FRt4/detection_loss‚3.441525760151091 %loss/total_loss‚43.37946583073298 epoch‚4 -loss/FBt4/detection_loss‚2.77710437503728 trainer/global_step‚4999 0loss/FRt2/assignment_loss‚2.6333506613969804 2loss/SRqqt2/assignment_loss‚1.4298313760757446‘ÕÆ_ÔÙ .loss/FBt2/detection_loss‚2.412618786096573 /loss/FRt1/assignment_loss‚2.743676861578768 3loss/FBt3/assignment_loss‚0.0014193959240553038 -loss/FRt1/detection_loss‚1.81983613119122 .loss/FRt4/detection_loss‚3.441525760151091 %loss/total_loss‚43.37946583073298 epoch‚4 -loss/FBt4/detection_loss‚2.77710437503728 trainer/global_step‚4999 0loss/FRt2/assignment_loss‚2.6333506613969804 2loss/SRqqt2/assignment_loss‚1.4298313760757446 1loss/FBt1/assignment_loss‚0.15569868052657515 /loss/FBt1/detection_loss‚0.7168657053403105 2loss/SRqqt1/assignment_loss‚1.5066013510028522 0loss/SRqqt1/detection_loss‚2.251903818299373 /loss/FBt3/detection_loss‚2.4520561893781028 0loss/FRt3/assignment_loss‚1.3051593426395864 /loss/FRt4/assignment_loss‚4.494871122496469 1loss/FBt2/assignment_loss‚0.17267373819558998 % _timestamp‚1.7885017948866193e+09 2loss/SRqqt4/assignment_loss‚1.7664857943143164 1loss/SRqqt3/detection_loss‚0.9972016336815851 _runtime‚5852.666835374 _step‚411 /loss/FRt2/detection_loss‚2.4412405961193144 2loss/SRqqt3/assignment_loss‚0.9810435458312969 0loss/FBt4/assignment_loss‚0.1117704096542928 /loss/FRt3/detection_loss‚1.4434261380092186 1loss/SRqqt2/detection_loss‚2.5790408754348757 0loss/SRqqt4/detection_loss‚2.744063542357513›i<Ž,lÕ‚(:á ¯¾éÔäÊÕn disk.md0p1.out‚ 629.30078125# disk.nvme1n1.out‚32444.07421875 disk.md0p1.in‚ 4.1796875 disk.nvme0n1.in‚ 939.5390625 disk.md0.in‚ 4.1796875 disk.sdb2.in‚ 0.6328125 disk.sda.out‚654.12548828125 disk.rbd0.out‚ 1.404296875! disk.sda2.out‚654.12548828125! disk.sdb2.out‚654.12548828125) proc.memory.availableMB‚ 1314.66796875# proc.memory.rssMB‚ 3045.43359375) disk./.usagePercent‚75.79055762467507 disk.sda1.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0$ memory_percent‚91.97590351104736 disk.md0.out‚ 629.30078125 network.sent‚ 1.234643e+07 network.recv‚ 1.947219e+06 disk.sdb.out‚654.12548828125 disk.sda2.in‚3.546875 disk.sda.in‚3.546875 cpu‚25.170567842033236 disk.sdb.in‚ 0.6328125 disk.rbd2.out‚ 1.404296875 disk.rbd0.in‚0 disk.nvme0n1.out‚ 54686.09375 disk.nvme1n1.in‚ 743.53125 disk.rbd2.in‚0 disk.dm-0.in‚ 1683.0703125 disk.dm-0.out‚87130.16796875* proc.memory.percent‚18.587851524353027$ disk./.usageGB‚2710.637451171875 disk.sda1.out‚0 proc.cpu.threads‚17Áô£“Ö‚(:ˆ ¯¾éÔÒÓÂn gpu.0.gpu‚31.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.597' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.89925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·¯t¡’ׂ(:‡ ¾¾éÔèÞ€m gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.118' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.0295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g3}GfØ‚(:Û ¾¾éÔÏ°œn disk.sdb.out‚655.9306640625 disk.sda2.in‚3.546875 disk.nvme1n1.in‚ 743.53125* proc.memory.percent‚18.587851524353027 disk.sdb2.in‚ 0.6328125 disk.rbd0.in‚0# memory_percent‚91.9774055480957 disk.sda.in‚3.546875 disk.sdb1.in‚0 disk.dm-0.out‚87443.53515625' proc.memory.availableMB‚ 1314.421875) disk./.usagePercent‚75.77581415264095 disk.sda1.out‚0 disk.rbd0.out‚ 1.416015625 disk.md0p1.out‚ 631.0390625 disk.dm-0.in‚ 1683.0703125 disk.sdb.in‚ 0.6328125 disk.sda2.out‚655.9306640625# proc.memory.rssMB‚ 3045.43359375 disk.rbd2.in‚0 disk.rbd2.out‚ 1.482421875 disk.md0.out‚ 631.0390625$ disk./.usageGB‚2710.110153198242 disk.sdb2.out‚655.9306640625 disk.sdb1.out‚0 disk.md0p1.in‚ 4.1796875 network.sent‚ 1.2441432e+07 network.recv‚ 1.953097e+06 disk.sda1.in‚0 disk.nvme0n1.in‚ 939.5390625 disk.md0.in‚ 4.1796875 cpu‚25.170223245692064 proc.cpu.threads‚17 disk.sda.out‚655.9306640625" disk.nvme0n1.out‚ 54843.8671875# disk.nvme1n1.out‚32599.667968752­"“Ù‚(:ˆ ;éÔ¢–Ëm gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.825' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.70625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W”F·lÚ‚(:á ;éÔÌñ»n# disk.nvme1n1.out‚32599.79296875 network.recv‚ 1.956609e+06 disk.sda2.in‚3.546875 disk.sda.out‚657.40771484375 disk.md0p1.out‚ 632.4609375 disk.md0.out‚ 632.4609375 disk.sdb.out‚657.40771484375! disk.sda2.out‚657.40771484375 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚18.587851524353027 disk.md0.in‚ 4.1796875) disk./.usagePercent‚75.77581223275138% disk./.usageGB‚2710.1100845336914 disk.nvme0n1.in‚ 939.5390625 disk.nvme1n1.in‚ 743.53125( proc.memory.availableMB‚ 1314.6640625# proc.memory.rssMB‚ 3045.43359375 cpu‚25.169874201122763 disk.sdb.in‚ 0.6328125! disk.nvme0n1.out‚ 54843.953125! disk.sdb2.out‚657.40771484375 disk.md0p1.in‚ 4.1796875 disk.dm-0.in‚ 1683.0703125 proc.cpu.threads‚17 network.sent‚ 1.2446687e+07 disk.rbd2.out‚ 1.482421875 disk.sdb2.in‚ 0.6328125 disk.sda.in‚3.546875 disk.rbd0.in‚0 disk.rbd0.out‚ 1.478515625 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.dm-0.out‚87443.74609375$ memory_percent‚91.975927352905271´Æ2Û- _wandbruntime‚5908  _runtime‚5908{ËøÇÜÁ & lr-AdamW/pg1‚0.0007607620756882154 & lr-AdamW/pg2‚0.0007607620756882154 trainer/global_step‚5049 % _timestamp‚1.7885018511365485e+09 _runtime‚5908.913791468 _step‚412Tˆ§=ÌÝÆ & lr-AdamW/pg2‚0.0007607620756882154 trainer/global_step‚5049 % _timestamp‚1.7885018511365485e+09 _runtime‚5908.913791468 _step‚412 & lr-AdamW/pg1‚0.0007607620756882154œ­–2Þ- _wandbruntime‚5908  _runtime‚5908ÁeÜ·ú'ßôO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5766333226982 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 233698557 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6048736269824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚425873988779520 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127953467 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63344922 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚417534398796374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚431662872163328 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127979811 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6048505390592 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _step‚413 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚411768065569392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 233672211 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94073258 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63344924 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚469840245395968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚431661944188928 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5788883383808 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 % _timestamp‚1.7885018511376274e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚431662872163328 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5787973412352 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚619 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105718744 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚411768082608752 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127979811 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚475888726846464 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5767241201986 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚431661944188928 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105718744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚469840221455872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856.K.aÿ'àùO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚425873988779520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30727715 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚475888981665792 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚431661944188928 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105718744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚469840221455872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5766333226982 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 233698557 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6048736269824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚425873988779520 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127953467 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63344922 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚417534398796374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚431662872163328 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127979811 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6048505390592 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _step‚413 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚411768065569392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 233672211 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94073258 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63344924 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚469840245395968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚431661944188928 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5788883383808 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 % _timestamp‚1.7885018511376274e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚431662872163328 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5787973412352 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚619 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105718744 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚411768082608752 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127979811 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚475888726846464 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5767241201986 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚431745612722688 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚469930447160320 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚417615458678768 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105738904 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5790004857344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5767450500260 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚425955607865344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30733536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚431745612722688 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5768358475472 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚425955589862400 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30734152 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 YIDeviceStatsMonitor.on_train_batch_end/res¢‰îVerved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94090523 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94091141 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 $ _timestamp‚1.788501852197524e+09 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 _runtime‚5909.976567492 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233743574 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128004668 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63356989 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6049911182848 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚411847100203296 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127978298 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105738906 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640¹¬Ø'ã‚N TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5789094872576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233743574 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚431745612722688 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚469930447160320 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚417615458678768 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105738904 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5790004857344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5767450500260 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚425955607865344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30733536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚431745612722688 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5768358475472 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚425955589862400 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30734152 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94090523 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94091141 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 $ _timestamp‚1.788501852197524e+09 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 _runtime‚5909.976567492 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233743574 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128004668 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63356989 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6049911182848 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚411847100203296 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127978298 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105738906 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784žÂÊè2ä- _wandbruntime‚5909  _runtime‚5909£*·HYåÓ 1loss/SRqqt1/detection_loss‚2.7274397533872854 trainer/global_step‚5049 %loss/total_loss‚43.10037267531392 /loss/FRt1/detection_loss‚1.9810683963484113 /loss/FRt1/assignment_loss‚2.709369769221858 2loss/SRqqt1/assignment_loss‚1.6279672079760095 2loss/SRqqt3/assignment_loss‚1.0529691383051085 0loss/FBt1/assignment_loss‚0.1982464823361016 /loss/FBt3/detection_loss‚1.9473128199577332 0loss/FBt4/assignment_loss‚0.8714994118100549 /loss/FRt2/detection_loss‚2.5017539733430234 0loss/SRqqt4/detection_loss‚2.421846291598152 /loss/FBt1/detection_loss‚0.8776486842591014 epoch‚4 1loss/FBt2/assignment_loss‚0.33712835925991336 /loss/FRt2/assignment_loss‚3.113851358250874 1loss/SRqqt2/assignment_loss‚2.189598840636176 2loss/SRqqt4/assignment_loss‚1.3134584593422272 0loss/SRqqt2/detection_loss‚2.498436246369336 % _timestamp‚1.7885018522007408e+09 _step‚415 0loss/FRt4/assignment_loss‚3.0206946082737134 /loss/FRt4/detection_loss‚3.0676086622735728 .loss/FBt2/detection_loss‚2.405410793092516 0loss/FRt3/assignment_loss‚1.5557923746459625 1loss/SRqqt3/detection_loss‚0.9122587595615657 _runtime‚5909.977785527 0loss/FBt3/assignment_loss‚0.0870375359296034 .loss/FBt4/detection_loss‚2.295825272798538 /loss/FRt3/detection_loss‚1.3861494763370823T+ôk^æØ 1loss/SRqqt3/detection_loss‚0.9122587595615657 _runtime‚5909.977785527 0loss/FBt3/assignment_loss‚0.0870375359296034 .loss/FBt4/detection_loss‚2.295825272798538 /loss/FRt3/detection_loss‚1.3861494763370823 1loss/SRqqt1/detection_loss‚2.7274397533872854 trainer/global_step‚5049 %loss/total_loss‚43.10037267531392 /loss/FRt1/detection_loss‚1.9810683963484113 /loss/FRt1/assignment_loss‚2.709369769221858 2loss/SRqqt1/assignment_loss‚1.6279672079760095 2loss/SRqqt3/assignment_loss‚1.0529691383051085 0loss/FBt1/assignment_loss‚0.1982464823361016 /loss/FBt3/detection_loss‚1.9473128199577332 0loss/FBt4/assignment_loss‚0.8714994118100549 /loss/FRt2/detection_loss‚2.5017539733430234 0loss/SRqqt4/detection_loss‚2.421846291598152 /loss/FBt1/detection_loss‚0.8776486842591014 epoch‚4 1loss/FBt2/assignment_loss‚0.33712835925991336 /loss/FRt2/assignment_loss‚3.113851358250874 1loss/SRqqt2/assignment_loss‚2.189598840636176 2loss/SRqqt4/assignment_loss‚1.3134584593422272 0loss/SRqqt2/detection_loss‚2.498436246369336 % _timestamp‚1.7885018522007408e+09 _step‚415 0loss/FRt4/assignment_loss‚3.0206946082737134 /loss/FRt4/detection_loss‚3.0676086622735728 .loss/FBt2/detection_loss‚2.405410793092516 0loss/FRt3/assignment_loss‚1.5557923746459625Ÿ¦&QP“ç‚(:ˆ ܾéÔɈ‹n gpu.0.gpu‚28.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.701' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.17525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w«”‹hè‚(:Ý ܾéÔíÆŸn) disk./.usagePercent‚75.77652003204253 disk.rbd2.in‚0 disk.sda.out‚658.79736328125 disk.sda1.out‚0# proc.memory.rssMB‚ 3045.43359375 cpu‚25.16953110145943$ disk./.usageGB‚2710.135398864746 disk.sdb.out‚658.79736328125 disk.nvme0n1.in‚ 939.5390625! disk.sdb2.out‚658.79736328125 disk.sda.in‚3.546875 disk.sda1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚18.587851524353027 disk.nvme1n1.in‚ 743.53125 disk.md0.out‚ 633.796875$ memory_percent‚91.97735786437988 proc.cpu.threads‚17 network.recv‚ 1.961033e+06! disk.sda2.out‚658.79736328125 disk.md0.in‚ 4.1796875 network.sent‚ 1.2452658e+07 disk.sdb1.out‚0 disk.md0p1.in‚ 4.1796875 disk.dm-0.out‚87874.36328125( proc.memory.availableMB‚ 1314.4296875 disk.md0p1.out‚ 633.796875 disk.dm-0.in‚ 1683.0703125 disk.rbd0.out‚ 1.478515625 disk.sdb1.in‚0 disk.sda2.in‚3.546875# disk.nvme0n1.out‚55230.42578125 disk.nvme1n1.out‚ 32643.9375 disk.rbd2.out‚ 1.56982421875 disk.sdb.in‚ 0.6328125 disk.sdb2.in‚ 0.6328125¼Äâé‚(:’ ë¾éÔåøçl gpu.0.gpu‚10.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚126.497' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚31.624249999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û4`dê‚(:Ù ë¾éÔÉ˳m disk.sda2.out‚ 660.845703125 disk.rbd0.in‚0 network.recv‚ 1.965947e+06 disk.sdb.in‚ 0.6328125 disk.md0.in‚ 4.1796875 disk.sda.in‚3.546875 network.sent‚ 1.2562417e+07 disk.md0.out‚ 635.78515625 disk.rbd2.out‚ 1.56982421875 disk.rbd0.out‚ 1.5576171875 disk.sdb1.out‚0 disk.sda2.in‚3.546875 disk.nvme1n1.in‚ 743.53125 disk.sda1.out‚0* proc.memory.percent‚18.587851524353027 proc.cpu.threads‚17) disk./.usagePercent‚75.77745576489083 disk.md0p1.out‚ 635.78515625 disk.dm-0.in‚ 1683.0703125" disk.nvme1n1.out‚ 32644.1015625 disk.sdb2.out‚ 660.845703125$ memory_percent‚91.97750091552734 disk.md0p1.in‚ 4.1796875 disk.rbd2.in‚0& proc.memory.availableMB‚ 1314.40625 disk.sdb1.in‚0 disk.dm-0.out‚ 87923.46875 disk.sdb.out‚ 660.845703125 disk.nvme0n1.in‚ 939.5390625 disk.sdb2.in‚ 0.6328125# proc.memory.rssMB‚ 3045.43359375% disk./.usageGB‚2710.1688652038574" disk.nvme0n1.out‚ 55292.3671875 disk.sda.out‚ 660.845703125 disk.sda1.in‚0 cpu‚25.169195175451616CWk‘ë‚(:† ú¾éÔ°ül gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K®dì‚(:Ù ú¾éÔ²ïÜl disk.dm-0.in‚ 1683.0703125 disk.sdb.in‚ 0.6328125 disk.sdb.out‚662.19482421875 disk.rbd2.in‚0 proc.cpu.threads‚17 network.sent‚ 1.2568739e+07) disk./.usagePercent‚75.78358319914015 disk.sda.in‚3.546875 disk.sda1.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 637.078125! disk.sdb2.out‚662.19482421875 disk.sda.out‚662.19482421875 disk.nvme1n1.in‚ 743.53125! disk.nvme1n1.out‚ 32644.671875 disk.rbd2.out‚ 1.6328125* proc.memory.percent‚18.587851524353027 disk.sdb2.in‚ 0.6328125 disk.sda1.out‚0 disk.dm-0.out‚ 88164.78125! disk.sda2.out‚662.19482421875$ memory_percent‚91.97747707366943 disk.rbd0.in‚0! disk.nvme0n1.out‚ 55520.109375# proc.memory.rssMB‚ 3045.43359375$ disk./.usageGB‚2710.388011932373) proc.memory.availableMB‚ 1314.41015625 network.recv‚ 1.970757e+06 disk.md0p1.in‚ 4.1796875 disk.nvme0n1.in‚ 939.5390625 disk.md0.in‚ 4.1796875 disk.md0p1.out‚ 637.078125 disk.sda2.in‚3.546875 cpu‚25.168817684035105 disk.rbd0.out‚ 1.5576171875 disk.sdb1.in‚0k¥³³œí‚(:‘ ‰¿éÔÒ¢˜m gpu.0.gpu‚17.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚117.81' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.452499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØãÓEjî‚(:ß ‰¿éÔ¿™Ám disk.dm-0.in‚ 1683.0703125 network.recv‚ 1.974227e+06 disk.rbd0.in‚0! disk.nvme0n1.out‚ 55644.171875# disk.nvme1n1.out‚32644.79296875 disk.sda2.in‚3.546875( disk./.usagePercent‚75.7835826658375 disk.rbd0.out‚ 1.62060546875 disk.sdb1.out‚0 disk.dm-0.out‚88288.96484375 disk.md0.in‚ 4.1796875 disk.rbd2.out‚ 1.6328125 disk.sdb2.in‚ 0.6328125 disk.sdb.in‚ 0.6328125 disk.sdb.out‚663.85693359375! disk.sda2.out‚663.85693359375 disk.nvme1n1.in‚ 743.53125# proc.memory.rssMB‚ 3045.43359375 disk.rbd2.in‚0 disk.nvme0n1.in‚ 939.5390625 cpu‚25.168512809968707 disk.sda1.in‚0 disk.sdb1.in‚0 disk.md0p1.in‚ 4.1796875) proc.memory.availableMB‚ 1314.41015625 disk.md0.out‚ 638.671875 disk.sda.in‚3.546875 disk.sda1.out‚0 disk.md0p1.out‚ 638.671875$ memory_percent‚91.97747707366943* proc.memory.percent‚18.587851524353027 proc.cpu.threads‚17 network.sent‚ 1.2574129e+07% disk./.usageGB‚2710.3879928588867! disk.sdb2.out‚663.85693359375 disk.sda.out‚663.85693359375XѺ2ï- _wandbruntime‚5966  _runtime‚5966D`äXÆðÀ & lr-AdamW/pg1‚0.0007539385864176873 & lr-AdamW/pg2‚0.0007539385864176873 trainer/global_step‚5099 $ _timestamp‚1.788501908661544e+09 _runtime‚5966.438814676 _step‚416‚ž§|ËñÅ trainer/global_step‚5099 $ _timestamp‚1.788501908661544e+09 _runtime‚5966.438814676 _step‚416 & lr-AdamW/pg1‚0.0007539385864176873 & lr-AdamW/pg2‚0.0007539385864176873 §7"\2ò- _wandbruntime‚5966  _runtime‚5966;Ld)OóôO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5823118457094 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚435798985905152 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129224713 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6107484249600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 trainer/global_step‚5099 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚415718962335952 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚435799913879552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5844970808832 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚5966.440001101 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 235925113 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚94965210 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚415718945296592 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚480459014867968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚474351506678272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 _step‚417 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚435799913879552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚429954943070720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106726746 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31017668 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106726744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94965842 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 129198369 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63948174 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5822210482090 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚429954943070720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚474351530618368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 1 Ï:p« 29224713 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚429954925067776 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚421542080793046 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚632 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5844060837376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚630 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚480458760048640 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚429954925067776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5844970808832 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 235951459 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129198369 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31017038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6107253370368 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106726746 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚435798985905152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚421541155778682ªÉ sÿ'ôùO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106726746 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚435798985905152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚421541155778682 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5823118457094 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚435798985905152 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129224713 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6107484249600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 trainer/global_step‚5099 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚415718962335952 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚435799913879552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5844970808832 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚5966.440001101 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 235925113 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚94965210 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚415718945296592 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚480459014867968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚474351506678272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 _step‚417 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚435799913879552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚429954943070720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106726746 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31017668 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106726744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94965842 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 129198369 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63948174 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5822210482090 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚429954943070720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚474351530618368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129224713 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚429954925067776 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚421542080793046 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚632 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5844060837376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚630 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚480458760048640 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚429954925067776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5844970808832 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 235951459 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129198369 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31017038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6107253370368 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185¡Œý(T2õ- _wandbruntime‚5967  _runtime‚5967ä$X'öþM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6108648398336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235996476 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129249570 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5845182297600 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6108419629568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚415797979930496 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚415797962891136 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63960237 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚435881726451200 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235970104 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94982969 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 % _timestamp‚1.7885019097462316e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚435882654438912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚629 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚421621290646504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 235996476 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5845182297600 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚474441732382720 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚430036562156544 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94983598 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106746904 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚430036562156544 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94983598 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106746904 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235996476 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129249570 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5845182297600 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6108419629568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚415797979930496 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚415797962891136 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63960237 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚435881726451200 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235970104 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94982969 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 % _timestamp‚1.7885019097462316e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚435882654438912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚629 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚421621290646504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 235996476 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5845182297600 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚474441732382720 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B3 ê2ƒ- _wandbruntime‚6023  _runtime‚6023ô$ÑÆ„À _runtime‚ 6023.73899077 _step‚420 & lr-AdamW/pg1‚0.0007470676319463533 & lr-AdamW/pg2‚0.0007470676319463533 trainer/global_step‚5149 % _timestamp‚1.7885019659617531e+09A‚dØË…Å _step‚420 & lr-AdamW/pg1‚0.0007470676319463533 & lr-AdamW/pg2‚0.0007470676319463533 trainer/global_step‚5149 % _timestamp‚1.7885019659617531e+09 _runtime‚ 6023.73899077¤§Aß2†- _wandbruntime‚6023  _runtime‚6023À]:ú'‡ôO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚419669825023792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚425548837775354 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 % _timestamp‚1.7885019659627974e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚439936955595776 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5900148262400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚439936027621376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚439936955595776 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚485028752310784 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107734744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107734746 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚95858947 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚434035897361920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5878995712202 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5901058233856 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 130443271 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64551424 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6166189192192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130469615 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107734744 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚434035897361920 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚434035879358976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚419669842063152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 _runtime‚6023.740087105 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64551422 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107734746 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5900148262400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚609 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31307523 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 238178015 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5901058233856 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95858336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚478862791900672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6165960410112 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚425547912760990 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚485029005032960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31306914 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚478862815840768áT¸"ÿ'ˆùO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚425548837775354 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 % _timestamp‚1.7885019659627974e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚439936955595776 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5900148262400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚439936027621376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚439936955595776 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚485028752310784 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107734744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107734746 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚95858947 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚434035897361920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5878995712202 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5901058233856 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 130443271 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64551424 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6166189192192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130469615 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107734744 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚434035897361920 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚434035879358976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚419669842063152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 _runtime‚6023.740087105 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64551422 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107734746 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5900148262400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚609 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31307523 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 238178015 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5901058233856 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95858336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚478862791900672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6165960410112 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚425547912760990 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚485029005032960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31306914 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚478862815840768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚419669825023792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185¥Û8ì2‰- _wandbruntime‚6024  _runtime‚6024#ÐY‰lŠüM PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130494472 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223104 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238249378 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚478953041545216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 238223006 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107754904 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚425628047628812 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚434117516447744 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 238223006 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5879205010476 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚440019696155136 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚613 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5880112985688 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107754906 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130494472 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚478953017605120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚425628972643384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6167149785088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚434117498444800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31312709 _step‚422 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚611 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95876196 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64563487 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚434117516447744 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5902179707392 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130468102 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31313320 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚5149 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107754906 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5902179707392 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚434117498444800 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚485120167390208 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 _runtime‚6024.875778933\øP'‹N B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130494472 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚478953017605120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚425628972643384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6167149785088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚434117498444800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31312709 _step‚422 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚611 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95876196 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64563487 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚434117516447744 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5902179707392 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130468102 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31313320 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚5149 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107754906 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5902179707392 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚434117498444800 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚485120167390208 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 _runtime‚6024.875778933 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130494472 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223104 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238249378 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚478953041545216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 238223006 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107754904 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚425628047628812 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚434117516447744 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 238223006 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5879205010476 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚440019696155136 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚613 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5880112985688 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107754906 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640¦G¡GÄ2Œ- _wandbruntime‚6024  _runtime‚6024E'X¾XÒ 0loss/SRqqt1/detection_loss‚2.116838973501454 2loss/SRqqt4/assignment_loss‚1.2642625076075396 1loss/FBt3/assignment_loss‚0.17604294744219423 1loss/SRqqt3/detection_loss‚0.8803945419111455 trainer/global_step‚5149 epoch‚4 % _timestamp‚1.7885019670998032e+09 0loss/FRt3/assignment_loss‚1.5269611087001738 1loss/FBt2/assignment_loss‚0.40386126539694184 1loss/SRqqt2/detection_loss‚2.3753809476720877 /loss/FBt1/detection_loss‚0.5856975442277164 .loss/FBt3/detection_loss‚2.126001439988613 2loss/SRqqt2/assignment_loss‚1.6586585702567265 0loss/FBt4/assignment_loss‚0.4869139878552232 .loss/FRt1/detection_loss‚1.693972510654324 /loss/FRt2/detection_loss‚2.2273529946804045 _step‚423 0loss/FRt2/assignment_loss‚3.5246250046624077 /loss/FRt4/assignment_loss‚2.373248591423035 1loss/SRqqt1/assignment_loss‚1.115360498185391 /loss/FRt3/detection_loss‚1.8333057668365416 .loss/FRt4/detection_loss‚2.992589795589447 1loss/SRqqt4/detection_loss‚2.4457832435766855 $loss/total_loss‚39.3159620539576 /loss/FBt2/detection_loss‚1.9818259358406067 .loss/FRt1/assignment_loss‚2.26589060359531 2loss/SRqqt3/assignment_loss‚1.0273313399462043 .loss/FBt4/detection_loss‚2.072199010848999 _runtime‚6024.876841848 1loss/FBt1/assignment_loss‚0.16146292355842384-]6]Ž× 1loss/FBt1/assignment_loss‚0.16146292355842384 0loss/SRqqt1/detection_loss‚2.116838973501454 2loss/SRqqt4/assignment_loss‚1.2642625076075396 1loss/FBt3/assignment_loss‚0.17604294744219423 1loss/SRqqt3/detection_loss‚0.8803945419111455 trainer/global_step‚5149 epoch‚4 % _timestamp‚1.7885019670998032e+09 0loss/FRt3/assignment_loss‚1.5269611087001738 1loss/FBt2/assignment_loss‚0.40386126539694184 1loss/SRqqt2/detection_loss‚2.3753809476720877 /loss/FBt1/detection_loss‚0.5856975442277164 .loss/FBt3/detection_loss‚2.126001439988613 2loss/SRqqt2/assignment_loss‚1.6586585702567265 0loss/FBt4/assignment_loss‚0.4869139878552232 .loss/FRt1/detection_loss‚1.693972510654324 /loss/FRt2/detection_loss‚2.2273529946804045 _step‚423 0loss/FRt2/assignment_loss‚3.5246250046624077 /loss/FRt4/assignment_loss‚2.373248591423035 1loss/SRqqt1/assignment_loss‚1.115360498185391 /loss/FRt3/detection_loss‚1.8333057668365416 .loss/FRt4/detection_loss‚2.992589795589447 1loss/SRqqt4/detection_loss‚2.4457832435766855 $loss/total_loss‚39.3159620539576 /loss/FBt2/detection_loss‚1.9818259358406067 .loss/FRt1/assignment_loss‚2.26589060359531 2loss/SRqqt3/assignment_loss‚1.0273313399462043 .loss/FBt4/detection_loss‚2.072199010848999 _runtime‚6024.876841848§Ȇ/á‘‚(:† Ô¿éÔ¯ÿl gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚173.344' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áñ ªn‚(:ã Ô¿éÔ‘’¯m* proc.memory.percent‚18.587851524353027 disk.md0.in‚ 4.1796875 disk.rbd2.out‚ 1.69580078125 cpu‚25.16679835491655 network.sent‚ 1.2715214e+07) disk./.usagePercent‚75.77658125518796 disk.nvme1n1.in‚ 743.53125 disk.rbd2.in‚0 disk.sda1.out‚0 disk.rbd0.out‚ 1.68359375 disk.md0p1.in‚ 4.1796875$ memory_percent‚91.97943210601807 disk.dm-0.in‚ 1683.0703125# disk.nvme0n1.out‚56823.30859375 disk.rbd0.in‚0 disk.md0p1.out‚ 647.0390625 proc.cpu.threads‚17 network.recv‚ 1.996698e+06 disk.sda2.in‚3.546875! disk.sdb2.out‚672.53271484375 disk.sdb1.out‚0# proc.memory.rssMB‚ 3045.43359375! disk.sda2.out‚672.53271484375 disk.sdb2.in‚ 0.6328125 disk.sdb1.in‚0 disk.sdb.out‚672.53271484375 disk.nvme0n1.in‚ 939.5390625 disk.md0.out‚ 647.0390625) proc.memory.availableMB‚ 1314.08984375% disk./.usageGB‚2710.1375885009766 disk.dm-0.out‚ 89455.3046875# disk.nvme1n1.out‚32645.62109375 disk.sdb.in‚ 0.6328125 disk.sda.in‚3.546875 disk.sda.out‚672.53271484375 disk.sda1.in‚0ö [ˆ’‘‚(:‡ ã¿éÔþœ–m gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚183.258' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.8145 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%žjûd’‚(:Ù ã¿éÔô­øm disk.sda2.out‚673.9775390625 disk.sdb2.in‚ 0.6328125 disk.sdb1.in‚0 disk.md0p1.in‚ 4.1796875* proc.memory.percent‚18.587851524353027 network.recv‚ 2.001636e+06 disk.sdb.in‚ 0.6328125 disk.nvme0n1.in‚ 939.5390625 disk.sda.in‚3.546875 disk.sdb1.out‚0( proc.memory.availableMB‚ 1314.3359375 disk.nvme0n1.out‚ 57048.5625 disk.md0.in‚ 4.1796875 disk.rbd0.out‚ 1.68359375$ memory_percent‚91.97793006896973 proc.cpu.threads‚17 network.sent‚ 1.2809755e+07$ disk./.usageGB‚2710.262870788574 disk.rbd2.out‚ 1.69580078125 disk.sda1.in‚0 disk.dm-0.out‚89694.27734375 disk.nvme1n1.in‚ 743.53125 disk.rbd2.in‚0 disk.sdb2.out‚673.9775390625 disk.md0p1.out‚ 648.421875 disk.sda2.in‚3.546875 disk.sda.out‚673.9775390625 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1683.0703125# disk.nvme1n1.out‚32645.71484375 disk.md0.out‚ 648.421875 cpu‚25.166463641363432) disk./.usagePercent‚75.78008420038391 disk.sdb.out‚673.9775390625# proc.memory.rssMB‚ 3045.43359375Ÿ÷‚Ü“‚(:’ ò¿éԋ̪k gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚119.978' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.994499999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìÉð–g”‚(:Ü ò¿éÔæ’m network.sent‚ 1.2818267e+07 disk.sdb.out‚675.48193359375 disk.md0.in‚ 4.1796875 disk.sda1.out‚0 cpu‚25.166101355003583 disk.md0p1.out‚ 649.8671875 disk.nvme0n1.in‚ 939.5390625# disk.nvme0n1.out‚57282.30859375! disk.sdb2.out‚675.48193359375$ memory_percent‚91.97797775268555 disk.sdb1.in‚0 disk.md0p1.in‚ 4.1796875 disk.sdb.in‚ 0.6328125 disk.rbd2.in‚0 disk.sda.out‚675.48193359375 disk.rbd0.in‚0 disk.nvme1n1.in‚ 743.53125 proc.cpu.threads‚17 network.recv‚ 2.006527e+06$ disk./.usageGB‚2710.387996673584 disk.sda2.in‚3.546875 disk.md0.out‚ 649.8671875 disk.rbd0.out‚ 1.68359375' proc.memory.availableMB‚ 1314.328125 disk.sdb1.out‚0) disk./.usagePercent‚75.78358277249804! disk.sda2.out‚675.48193359375# disk.nvme1n1.out‚32645.94140625 disk.sda.in‚3.546875# proc.memory.rssMB‚ 3045.43359375* proc.memory.percent‚18.587851524353027 disk.rbd2.out‚ 1.69580078125 disk.sdb2.in‚ 0.6328125 disk.sda1.in‚0 disk.dm-0.in‚ 1683.0703125 disk.dm-0.out‚89928.25³Cò™“•‚(:ˆ ÀéÔ΀»m gpu.0.gpu‚31.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.039' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.00975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*Wjf–‚(:Û ÀéÔΫ¥n proc.cpu.threads‚17 network.sent‚ 1.2823529e+07 disk.sdb2.in‚ 0.6328125$ memory_percent‚91.97931289672852 disk.sdb1.out‚0 disk.sdb2.out‚677.4443359375 disk.sda.in‚3.546875 disk.rbd0.out‚ 1.68359375! disk.nvme0n1.out‚ 57569.953125# proc.memory.rssMB‚ 3045.43359375$ disk./.usageGB‚2710.638500213623 disk.sda1.in‚0 disk.md0p1.out‚ 651.7578125 disk.sdb.out‚677.4443359375 network.recv‚ 2.009962e+06 disk.sda.out‚677.4443359375 disk.sdb1.in‚0 disk.md0p1.in‚ 4.1796875 disk.sda2.out‚677.4443359375 disk.md0.out‚ 651.7578125 disk.rbd2.in‚0' proc.memory.availableMB‚ 1314.109375 disk.sda1.out‚0 disk.sdb.in‚ 0.6328125* proc.memory.percent‚18.587851524353027 cpu‚25.165805908239566 disk.dm-0.out‚90216.01171875 disk.nvme1n1.in‚ 743.53125 disk.md0.in‚ 4.1796875 disk.rbd2.out‚ 1.69580078125 disk.nvme0n1.in‚ 939.5390625# disk.nvme1n1.out‚32646.05859375( disk./.usagePercent‚75.7905869563214 disk.rbd0.in‚0 disk.dm-0.in‚ 1683.0703125 disk.sda2.in‚3.546875A£¿×2—- _wandbruntime‚6081  _runtime‚6081ïðÆ Å˜¿ & lr-AdamW/pg1‚0.0007401506441708272 & lr-AdamW/pg2‚0.0007401506441708272 trainer/global_step‚5199 $ _timestamp‚1.788502023807251e+09 _runtime‚ 6081.58450303 _step‚424—vp™Ê™Ä & lr-AdamW/pg1‚0.0007401506441708272 & lr-AdamW/pg2‚0.0007401506441708272 trainer/global_step‚5199 $ _timestamp‚1.788502023807251e+09 _runtime‚ 6081.58450303 _step‚424¨ÃWó©2š- _wandbruntime‚6081  _runtime‚6081ÅàŒÊ›ôO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65154672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31596579 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6224772406784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚429555594757662 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚429554669743298 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131714517 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5956235687424 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚611 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 240430917 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5934872967310 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚444073069337600 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚613 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚438116833650176 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212004864 `LDeviceStatsMonitor.o!,Cán_train_batch_start/allocated_bytes.large_pool.allocated‚438116851653120 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚483374101063168 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6224984411648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131714517 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108742744 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5957145658880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚96751864 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96751251 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚423620721790352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108742744c<-ÿ'œùO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65154672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31596579 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6224772406784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚429555594757662 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚429554669743298 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131714517 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5956235687424 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚611 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 240430917 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5934872967310 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚444073069337600 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚613 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚438116833650176 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212004864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚438116851653120 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚483374101063168 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6224984411648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131714517 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108742744 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5957145658880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚96751864 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96751251 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚423620721790352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108742744 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615©HŠ9…2- _wandbruntime‚6082  _runtime‚6082:’¹{'žþM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚429635729625692 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚438198470738944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 % _timestamp‚1.7885020247561495e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚96769659 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 240475908 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚489690486163968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚645 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6226159396352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6225922238976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31602920 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108762906 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108762904 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚444155809883648 _runtime‚6082.534092407 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚489690225066496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 240502280 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131713004 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚438198452736000 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚438198470738944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚647 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚429634804611120 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5935990240796 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108762906 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131739374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _step‚426 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108762904 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚423699739384896 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5957357147648 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚444156737871360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 240475908 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 131713004 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240502280 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96769012 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65166739 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚483464326767616 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚483464302827520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5958267132416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65166737 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 9099847681åýÇŸƒN REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31602275 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚423699722345536 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5935082265584 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 trainer/global_step‚5199 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚444156737871360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65166739 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚483464326767616 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚483464302827520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5958267132416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65166737 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚429635729625692 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚438198470738944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 % _timestamp‚1.7885020247561495e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚96769659 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 240475908 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚489690486163968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚645 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6226159396352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6225922238976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31602920 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108762906 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108762904 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚444155809883648 _runtime‚6082.534092407 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚489690225066496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 240502280 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131713004 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚438198452736000 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚438198470738944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚647 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚429634804611120 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5935990240796 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108762906 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131739374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _step‚426 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108762904 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚423699739384896 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5957357147648 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚444156737871360 HÕÏq‘B :DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 240475908 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 131713004 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240502280 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96769012 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚442197787941376 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 242683819 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 242710165 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 _runtime‚6139.035471776 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109750746 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚433561426725606 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚97645193 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31887880 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6283653612032 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6283433218560 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6013233083904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚427571601517552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚448210111053824 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚433562351739970 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚442197787941376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132933075 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚429 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚442197805944320 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65757922 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚448210111053824 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132959419 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5989842247414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132959419 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚494169039897600 trainer/global_step‚5249 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚448210111053824 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚433562351739970 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚442197787941376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132933075 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚429 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚442197805944320 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65757922 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚448210111053824 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132959419 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5989842247414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132959419 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚494169039897600 trainer/global_step‚5249 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚442197787941376 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 242683819 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 242710165 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 _runtime‚6139.035471776 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109750746 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚433561426725606 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚97645193 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31887880 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6283653612032 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6283433218560 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6013233083904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚427571601517552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109770906 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚427650619112096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _step‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚448293779587584 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚494260184001024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6013444572672 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97663043 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132957906 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132957906 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚612 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚433642486608000 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65769989 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6284595951104 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚433641561593428 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 132984276 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚487975588049920 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 % _timestamp‚1.7885020823016944e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 trainer/global_step‚5249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚448293779587584 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚442279425030144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109770904 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚442279407027200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31893668 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6284826817024 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5991867495904 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚6140.079665627 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 242755182 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚448292851599872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚442279425030144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31893056 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6014354557440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132984276 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97663657 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚494260438807040 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65769987 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 242728810 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚487975611990016 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5990959520692 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 242728810 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6013444572672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109770906 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚427650602072736^-ÉÐ '³ƒN RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109770906 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚427650602072736 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109770906 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚427650619112096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _step‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚448293779587584 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚494260184001024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6013444572672 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97663043 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132957906 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132957906 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚612 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚433642486608000 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65769989 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6284595951104 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚433641561593428 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 132984276 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚487975588049920 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 % _timestamp‚1.7885020823016944e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 trainer/global_step‚5249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚448293779587584 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚442279425030144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109770904 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚442279407027200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31893668 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6284826817024 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5991867495904 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚6140.079665627 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 242755182 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚448292851599872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚442279425030144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31893056 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6014354557440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132984276 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97663657 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚494260438807040 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65769987 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 242728810 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚487975611990016 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5990959520692 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 242728810 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6013444572672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360®h-è2´- _wandbruntime‚6140  _runtime‚6140C·æ~\µÖ 0loss/FRt4/assignment_loss‚3.1692615697781243 2loss/SRqqt2/assignment_loss‚1.8922074237079531 /loss/FBt2/detection_loss‚2.8888337075710298 .loss/FRt2/detection_loss‚2.158632749648288 /loss/FRt1/detection_loss‚1.6364880442515843 .loss/FRt3/detection_loss‚1.495977722108364 0loss/SRqqt3/detection_loss‚1.049075529697018 _runtime‚6140.080779142 0loss/FRt3/assignment_loss‚1.5775657817721367 1loss/FBt2/assignment_loss‚0.34395107205273234 .loss/FBt4/detection_loss‚2.203450953960419 0loss/FRt2/assignment_loss‚3.1315766850183175 /loss/FBt1/detection_loss‚0.8532260711140492 epoch‚4 trainer/global_step‚5249 2loss/SRqqt4/assignment_loss‚1.3673226108153662 .loss/FBt3/detection_loss‚1.715511728078127 %loss/total_loss‚41.20524784029724 0loss/FRt1/assignment_loss‚2.1907787534925673 2loss/SRqqt3/assignment_loss‚1.1049737080316477 1loss/FBt1/assignment_loss‚0.20713787302060732 1loss/FBt4/assignment_loss‚0.23339187291858252 1loss/SRqqt4/detection_loss‚2.2986702501773832 .loss/FRt4/detection_loss‚3.130352407693863 _step‚431 % _timestamp‚1.7885020823036113e+09 2loss/SRqqt1/assignment_loss‚1.2340332919901067 0loss/FBt3/assignment_loss‚0.1233209875226744 1loss/SRqqt1/detection_loss‚2.6086367626081812 1loss/SRqqt2/detection_loss‚2.5908702832681163G3Ìïa¶Û .loss/FRt2/detection_loss‚2.158632749648288 /loss/FRt1/detection_loss‚1.6364880442515843 .loss/FRt3/detection_loss‚1.495977722108364 0loss/SRqqt3/detection_loss‚1.049075529697018 _runtime‚6140.080779142 0loss/FRt3/assignment_loss‚1.5775657817721367 1loss/FBt2/assignment_loss‚0.34395107205273234 .loss/FBt4/detection_loss‚2.203450953960419 0loss/FRt2/assignment_loss‚3.1315766850183175 /loss/FBt1/detection_loss‚0.8532260711140492 epoch‚4 trainer/global_step‚5249 2loss/SRqqt4/assignment_loss‚1.3673226108153662 .loss/FBt3/detection_loss‚1.715511728078127 %loss/total_loss‚41.20524784029724 0loss/FRt1/assignment_loss‚2.1907787534925673 2loss/SRqqt3/assignment_loss‚1.1049737080316477 1loss/FBt1/assignment_loss‚0.20713787302060732 1loss/FBt4/assignment_loss‚0.23339187291858252 1loss/SRqqt4/detection_loss‚2.2986702501773832 .loss/FRt4/detection_loss‚3.130352407693863 _step‚431 % _timestamp‚1.7885020823036113e+09 2loss/SRqqt1/assignment_loss‚1.2340332919901067 0loss/FBt3/assignment_loss‚0.1233209875226744 1loss/SRqqt1/detection_loss‚2.6086367626081812 1loss/SRqqt2/detection_loss‚2.5908702832681163 0loss/FRt4/assignment_loss‚3.1692615697781243 2loss/SRqqt2/assignment_loss‚1.8922074237079531 /loss/FBt2/detection_loss‚2.8888337075710298¯Ÿoh·‚(:Ý ÌÀéÔÇŒžn$ memory_percent‚91.97971820831299 network.sent‚ 1.2952457e+07 disk.rbd2.out‚ 1.69580078125 disk.sda.in‚3.546875 disk.sda1.in‚0 disk.sdb1.in‚0 disk.sdb.in‚ 0.6328125 disk.nvme1n1.in‚ 743.53125! disk.nvme1n1.out‚ 32721.078125 disk.md0p1.in‚ 4.1796875# proc.memory.rssMB‚ 3045.43359375 disk.rbd0.in‚0! disk.sdb2.out‚685.34033203125% disk./.usageGB‚2710.1375465393066 disk.md0.out‚ 659.3203125 disk.rbd2.in‚0 disk.sda1.out‚0* proc.memory.percent‚18.587851524353027 disk.sdb2.in‚ 0.6328125 disk.sdb1.out‚0 disk.md0p1.out‚ 659.3203125) proc.memory.availableMB‚ 1314.04296875 cpu‚25.16419750152623 proc.cpu.threads‚17 network.recv‚ 2.031901e+06 disk.dm-0.out‚ 91328.34375! disk.nvme0n1.out‚ 58607.265625 disk.sda.out‚685.34033203125 disk.rbd0.out‚ 1.68359375) disk./.usagePercent‚75.77658008192212 disk.dm-0.in‚ 1683.0703125 disk.sdb.out‚685.34033203125 disk.sda2.in‚3.546875! disk.sda2.out‚685.34033203125 disk.nvme0n1.in‚ 939.5390625 disk.md0.in‚ 4.1796875»*a‘¸‚(:† ÌÀéÔªæ˜o gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.77' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[:‘¹‚(:† ÛÀéÔÖÏûl gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.168' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞUâlº‚(:á ÛÀéÔn disk.sda2.in‚3.546875# disk.nvme1n1.out‚32721.17578125 disk.rbd0.in‚0 disk.sdb1.out‚0 cpu‚25.163922487666866 proc.cpu.threads‚17 disk.nvme0n1.in‚ 939.5390625 disk.md0.in‚ 4.1796875 disk.md0.out‚ 660.78515625 disk.sda1.in‚0* proc.memory.percent‚18.587851524353027 disk.sdb1.in‚0 disk.md0p1.out‚ 660.78515625 disk.dm-0.in‚ 1683.0703125 disk.sdb.out‚686.85791015625( proc.memory.availableMB‚ 1314.0390625' disk./.usagePercent‚75.780087826842# disk.nvme0n1.out‚58607.33984375! disk.sdb2.out‚686.85791015625 disk.dm-0.out‚ 91328.515625% disk./.usageGB‚2710.2630004882812 disk.sdb2.in‚ 0.6328125 disk.sda.in‚3.546875# proc.memory.rssMB‚ 3045.43359375 disk.nvme1n1.in‚ 743.53125 disk.rbd2.in‚0 disk.sda.out‚686.85791015625 disk.md0p1.in‚ 4.1796875 disk.sdb.in‚ 0.6328125 network.sent‚ 1.3062098e+07! disk.sda2.out‚686.85791015625 disk.rbd2.out‚ 1.69580078125 network.recv‚ 2.036678e+06 disk.sda1.out‚0 disk.rbd0.out‚ 1.68359375# memory_percent‚91.9797420501709.”m§“»‚(:ˆ êÀéÔ˜¤¿m gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.109' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.52725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q»äh¼‚(:Ý êÀéÔÎŒ¢n disk.sda1.in‚0 disk.md0.in‚ 4.1796875 disk.rbd2.in‚0 disk.sda.out‚688.8115234375# disk.nvme1n1.out‚32902.52734375 disk.dm-0.in‚ 1683.0703125) proc.memory.availableMB‚ 1314.04296875# proc.memory.rssMB‚ 3045.43359375 network.recv‚ 2.041222e+06 disk.sda2.in‚3.546875 disk.sda2.out‚688.8115234375 disk.md0.out‚ 662.671875 disk.sda.in‚3.546875* proc.memory.percent‚18.587851524353027 proc.cpu.threads‚17 disk.sdb2.out‚688.8115234375% disk./.usageGB‚2710.3872718811035 disk.rbd0.out‚ 1.68359375 disk.sdb1.in‚0 disk.sdb.in‚ 0.6328125 disk.sdb1.out‚0 disk.sdb.out‚688.8115234375# disk.nvme0n1.out‚58922.54296875 disk.rbd2.out‚ 1.69580078125 disk.md0p1.out‚ 662.671875 disk.dm-0.out‚ 91825.0703125 disk.sda1.out‚0 disk.rbd0.in‚0 disk.md0p1.in‚ 4.1796875 disk.sdb2.in‚ 0.6328125 network.sent‚ 1.3068374e+07 disk.nvme0n1.in‚ 939.5390625 disk.nvme1n1.in‚ 743.53125$ memory_percent‚91.97971820831299 cpu‚25.16360482154654) disk./.usagePercent‚75.78356250699693á:¢c½‚(:Ø ùÀéÔ„ôÇn disk.sda.in‚3.546875% disk./.usageGB‚2710.6382904052734 disk.dm-0.in‚ 1683.0703125 disk.sda.out‚690.2880859375# proc.memory.rssMB‚ 3045.43359375* proc.memory.percent‚18.587851524353027 disk.sdb1.in‚0 disk.md0p1.in‚ 4.1796875 disk.sdb.in‚ 0.6328125 disk.sda1.in‚0 disk.sdb1.out‚0# disk.nvme0n1.out‚58922.59765625# disk.nvme1n1.out‚32902.64453125 disk.rbd2.out‚ 1.69580078125 disk.nvme0n1.in‚ 939.5390625 cpu‚25.163248015614695) disk./.usagePercent‚75.79058108999213 disk.sda2.in‚3.546875 disk.sda2.out‚690.2880859375 disk.nvme1n1.in‚ 743.53125 disk.md0.in‚ 4.1796875 disk.md0.out‚ 664.09375 disk.sdb2.in‚ 0.6328125 network.sent‚ 1.3073866e+07 disk.rbd0.in‚0 disk.rbd0.out‚ 1.68359375 disk.dm-0.out‚ 91825.2421875 disk.sda1.out‚0# memory_percent‚91.9797420501709 network.recv‚ 2.0447e+06 disk.md0p1.out‚ 664.09375 disk.sdb.out‚690.2880859375 disk.sdb2.out‚690.2880859375( proc.memory.availableMB‚ 1314.0390625 proc.cpu.threads‚17 disk.rbd2.in‚0N]‘N“¾‚(:ˆ ùÀéÔÐÎŽo gpu.0.gpu‚54.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üÿ]s2¿- _wandbruntime‚6196  _runtime‚6196ùíFœÇÀÁ _step‚432 & lr-AdamW/pg1‚0.0007261843439596049 & lr-AdamW/pg2‚0.0007261843439596049 trainer/global_step‚5299 % _timestamp‚1.7885021389209309e+09 _runtime‚6196.698316061תÌÁÆ trainer/global_step‚5299 % _timestamp‚1.7885021389209309e+09 _runtime‚6196.698316061 _step‚432 & lr-AdamW/pg1‚0.0007261843439596049 & lr-AdamW/pg2‚0.0007261843439596049°Ìp2Â- _wandbruntime‚6196  _runtime‚6196Ô (éà ÃóO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6045719502522 $ _timestamp‚1.788502138922445e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚98539815 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32179283 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6068410537472 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244963067 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 239267840 VGDeviceStatsMonitor.on_train_batch_start/requestÔ†ÒÓed_bytes.small_pool.peak‚ 1187728393 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110758746 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134177977 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6068410537472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚452348080744448 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚452347152770048 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚431522481244752 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚98539815 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32179283 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6068410537472 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244963067 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 239267840 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110758746 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134177977 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6068410537472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚452348080744448 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚452347152770048 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚431522481244752 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 245008084 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244981712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6047744751012 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚446360379321344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚492486897212416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚431601481799936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 % _timestamp‚1.7885021400773659e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 245008084 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 134229178 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚98558177 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚452429893316096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _runtime‚6197.856625875 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚498830432236544 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6069531997696 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 134202808 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚446360361318400 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110778904 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚446360361318400 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6070441982464 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66373237 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32184317 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚498830171139072 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚5299 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98557554 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚431601498839296 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚434 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 245008084 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 134229178 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚98558177 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚452429893316096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _runtime‚6197.856625875 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚498830432236544 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6069531997696 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 134202808 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚446360361318400 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110778904 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚446360361318400 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6070441982464 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66373237 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32184317 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MCDeviceStatsMonitor.on_train_batch_end/allocatž~yion.small_pool.current‚26370 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚498830171139072 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚5299 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98557554 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚431601498839296 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚434 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 245008084 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244981712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6047744751012 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640²&,Š2È- _wandbruntime‚6197  _runtime‚6197QÓ‰-[ÉÕ 0loss/FRt4/assignment_loss‚3.5687167601925984 1loss/SRqqt3/detection_loss‚1.0698860903393548 epoch‚4 _step‚435 1loss/SRqqt2/assignment_loss‚1.192043530504878 1loss/FBt4/assignment_loss‚0.09728084300544045 /loss/FRt2/detection_loss‚2.3401538536560778 1loss/SRqqt1/detection_loss‚1.9284015049537022 trainer/global_step‚5299 % _timestamp‚1.7885021400810692e+09 0loss/FBt3/assignment_loss‚0.3184833208152327 /loss/FRt1/detection_loss‚1.5301458760396516 %loss/total_loss‚40.60831387197745 2loss/SRqqt1/assignment_loss‚1.5959070432517264 2loss/SRqqt3/assignment_loss‚0.9817223207814232 _runtime‚6197.858248002 0loss/FRt2/assignment_loss‚2.7346692024445045 2loss/SRqqt4/assignment_loss‚1.7917558363190405 .loss/FRt4/detection_loss‚3.010993699942316 0loss/SRqqt4/detection_loss‚2.594213331187213 /loss/FBt1/detection_loss‚0.7128542954586956 /loss/FRt1/assignment_loss‚1.860507304018194 0loss/FBt2/assignment_loss‚0.3375958593096584 .loss/FBt4/detection_loss‚2.497158482670784 0loss/FRt3/assignment_loss‚1.6808899398577415 0loss/FBt1/assignment_loss‚0.1408772242072464 /loss/FBt3/detection_loss‚1.9050182146685464 /loss/FRt3/detection_loss‚1.6704317804867939 1loss/SRqqt2/detection_loss‚2.2850201486087425 /loss/FBt2/detection_loss‚2.7635874092578887dG(`ÊÚ /loss/FRt1/assignment_loss‚1.860507304018194 0loss/FBt2/assignment_loss‚0.3375958593096584 .loss/FBt4/detection_loss‚2.497158482670784 0loss/FRt3/assignment_loss‚1.6808899398577415 0loss/FBt1/assignment_loss‚0.1408772242072464 /loss/FBt3/detection_loss‚1.9050182146685464 /loss/FRt3/detection_loss‚1.6704317804867939 1loss/SRqqt2/detection_loss‚2.2850201486087425 /loss/FBt2/detection_loss‚2.7635874092578887 0loss/FRt4/assignment_loss‚3.5687167601925984 1loss/SRqqt3/detection_loss‚1.0698860903393548 epoch‚4 _step‚435 1loss/SRqqt2/assignment_loss‚1.192043530504878 1loss/FBt4/assignment_loss‚0.09728084300544045 /loss/FRt2/detection_loss‚2.3401538536560778 1loss/SRqqt1/detection_loss‚1.9284015049537022 trainer/global_step‚5299 % _timestamp‚1.7885021400810692e+09 0loss/FBt3/assignment_loss‚0.3184833208152327 /loss/FRt1/detection_loss‚1.5301458760396516 %loss/total_loss‚40.60831387197745 2loss/SRqqt1/assignment_loss‚1.5959070432517264 2loss/SRqqt3/assignment_loss‚0.9817223207814232 _runtime‚6197.858248002 0loss/FRt2/assignment_loss‚2.7346692024445045 2loss/SRqqt4/assignment_loss‚1.7917558363190405 .loss/FRt4/detection_loss‚3.010993699942316 0loss/SRqqt4/detection_loss‚2.594213331187213 /loss/FBt1/detection_loss‚0.7128542954586956³t¬\’Ë‚(:‡ ˆÁéÔû“îm gpu.0.gpu‚57.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚176.738' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.1845 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É—‚£nÌ‚(:ã ˆÁéÔ¼÷Ín network.sent‚ 1.3077665e+07 disk.sdb1.in‚0$ disk./.usageGB‚2710.638240814209 disk.nvme0n1.in‚ 939.5390625# disk.nvme0n1.out‚59132.32421875" disk.nvme1n1.out‚ 33109.3828125$ memory_percent‚91.97995662689209 disk.dm-0.in‚ 1683.0703125 disk.dm-0.out‚92241.70703125 disk.sda2.in‚3.546875 disk.nvme1n1.in‚ 743.53125 disk.sda.in‚3.546875 disk.sda1.in‚0 disk.md0p1.in‚ 4.1796875 disk.md0.out‚ 665.6328125 disk.sdb2.in‚ 0.6328125# proc.memory.rssMB‚ 3045.43359375 cpu‚25.162935065393803 disk.sdb.out‚691.90771484375 disk.md0.in‚ 4.1796875 disk.rbd0.in‚0) proc.memory.availableMB‚ 1314.00390625* proc.memory.percent‚18.587851524353027) disk./.usagePercent‚75.79057970340521 disk.rbd2.in‚0 disk.sda.out‚691.90771484375 disk.sda1.out‚0 disk.sdb.in‚ 0.6328125! disk.sda2.out‚691.90771484375 disk.rbd2.out‚ 1.69580078125 disk.rbd0.out‚ 1.68359375 disk.sdb1.out‚0 network.recv‚ 2.049288e+06 disk.md0p1.out‚ 665.6328125! disk.sdb2.out‚691.90771484375 proc.cpu.threads‚17\ÊÏ“Í‚(:ˆ —ÁéÔð»ºm gpu.0.gpu‚46.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.207' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.55175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 gÓÊ[΂(:Ð —ÁéԥǢn disk.rbd2.in‚0 disk.rbd2.out‚ 1.75830078125 disk.sda1.in‚0* proc.memory.percent‚18.587851524353027$ disk./.usageGB‚2710.638240814209# disk.nvme1n1.out‚33109.51953125 disk.md0.out‚ 667.19140625 disk.sdb2.in‚ 0.6328125 disk.sda.in‚3.546875 disk.sdb1.out‚0 disk.md0p1.out‚ 667.19140625! memory_percent‚91.97998046875 disk.nvme1n1.in‚ 743.53125 disk.rbd0.out‚ 1.68359375 disk.sdb1.in‚0 disk.dm-0.in‚ 1683.0703125# proc.memory.rssMB‚ 3045.43359375 cpu‚25.162626924081824 proc.cpu.threads‚17 network.sent‚ 1.318634e+07 network.recv‚ 2.053477e+06 disk.sda.out‚ 693.544921875 disk.md0p1.in‚ 4.1796875 proc.memory.availableMB‚1314 disk.sda2.out‚ 693.544921875 disk.rbd0.in‚0 disk.md0.in‚ 4.1796875 disk.sdb.in‚ 0.6328125 disk.sdb2.out‚ 693.544921875 disk.sda2.in‚3.546875 disk.nvme0n1.in‚ 939.5390625 disk.sda1.out‚0 disk.dm-0.out‚92346.79296875) disk./.usagePercent‚75.79057970340521 disk.sdb.out‚ 693.544921875" disk.nvme0n1.out‚ 59237.2734375 õXè’Ï‚(:‡ ¦ÁéÔ›¬˜m gpu.0.gpu‚31.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.774' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.9435 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥ "/cЂ(:Ø ¦ÁéԜׂn disk.md0.in‚ 4.1796875! disk.nvme0n1.out‚ 59409.515625 network.sent‚ 1.3190558e+07 network.recv‚ 2.058639e+06 disk.sda.out‚695.38818359375 disk.sda1.in‚0 disk.dm-0.out‚92562.87890625 disk.sdb.out‚695.38818359375! memory_percent‚91.97998046875 disk.sda1.out‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 1683.0703125 disk.md0.out‚ 668.95703125 proc.memory.availableMB‚1314$ disk./.usageGB‚2710.110683441162 disk.rbd0.in‚0 disk.rbd0.out‚ 1.68359375 disk.sdb.in‚ 0.6328125 disk.nvme0n1.in‚ 939.5390625 disk.rbd2.in‚0# proc.memory.rssMB‚ 3045.43359375* proc.memory.percent‚18.587851524353027 disk.sda.in‚3.546875 disk.md0p1.in‚ 4.1796875 cpu‚25.162319445176365 proc.cpu.threads‚17 disk.md0p1.out‚ 668.95703125 disk.sda2.in‚3.546875! disk.sdb2.out‚695.38818359375# disk.nvme1n1.out‚33153.36328125 disk.sdb2.in‚ 0.6328125) disk./.usagePercent‚75.77582897845491 disk.rbd2.out‚ 1.75830078125 disk.sdb1.in‚0! disk.sda2.out‚695.38818359375 disk.nvme1n1.in‚ 743.53125þÿª&2Ñ- _wandbruntime‚6254  _runtime‚6254ÅK‰oÆÒÀ & lr-AdamW/pg1‚0.0007191379420799397 & lr-AdamW/pg2‚0.0007191379420799397 trainer/global_step‚5349 % _timestamp‚1.7885021963122602e+09 _runtime‚ 6254.08953152 _step‚436C/ÀËÓÅ trainer/global_step‚5349 % _timestamp‚1.7885021963122602e+09 _runtime‚ 6254.08953152 _step‚436 & lr-AdamW/pg1‚0.0007191379420799397 & lr-AdamW/pg2‚0.0007191379420799397´bfù2Ô- _wandbruntime‚6254  _runtime‚6254LUœlú'ÕôO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚450359696523776 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6125407933952 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6102504732634 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135449223 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚626 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 trainer/global_step‚5349 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚441574940690222 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66964422 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111766744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚437 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚6254.090316454 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 247189623 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚496907932790272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135422879 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚450359714526720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6124497962496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚99436175 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚450359696523776 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135422879 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6401116067840 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135449223 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6124497962496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240139264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚435473343932592 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 247215969 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216199168 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚503308832658944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚456484194486272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6125407933952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32471127 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚450359714526720 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚456484194486272 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚441575865704586 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚435473360971952 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111766744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6101596757630 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111766746 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚503309072798208 »‚àÿ'ÖùO [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135422879 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6401116067840 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135449223 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6124497962496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240139264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚435473343932592 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 247215969 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216199168 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚503308832658944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚456484194486272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6125407933952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32471127 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚450359714526720 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚456484194486272 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚441575865704586 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚435473360971952 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111766744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6101596757630 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111766746 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚503309072798208 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚450359696523776 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6125407933952 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6102504732634 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135449223 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚626 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 trainer/global_step‚5349 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚441574940690222 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66964422 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111766744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚437 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚6254.090316454 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 247189623 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚496907932790272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135422879 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚450359714526720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6124497962496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚99436175 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚450359696523776 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615µi:pjRׂ(:† µÁéÔáêÿm gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAlwx^ ?located‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.168' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05)âLhØ‚(:Ý µÁéÔЕ¬n network.recv‚ 2.062054e+06 disk.sdb.in‚ 0.6328125 disk.rbd2.in‚0 disk.sdb1.out‚0$ memory_percent‚91.98002815246582( proc.memory.availableMB‚ 1313.9921875! disk.sdb2.out‚696.94580078125 disk.sda1.out‚0* proc.memory.percent‚18.587851524353027 proc.cpu.threads‚17% disk./.usageGB‚2710.0660133361816 disk.sdb.out‚696.94580078125! disk.nvme0n1.out‚ 59552.515625 disk.rbd0.in‚0 network.sent‚ 1.3195801e+07 disk.nvme1n1.in‚ 743.53125 disk.rbd2.out‚ 1.7822265625 disk.sdb2.in‚ 0.6328125 disk.md0p1.out‚ 670.44140625 disk.rbd0.out‚ 1.68359375 disk.md0p1.in‚ 4.1796875! disk.sda2.out‚696.94580078125 disk.nvme0n1.in‚ 939.5390625 disk.md0.in‚ 4.1796875 disk.dm-0.in‚ 1683.0703125 disk.dm-0.out‚ 92706.0625 cpu‚25.162008633698377 disk.sda2.in‚3.546875 disk.sda1.in‚0 disk.sdb1.in‚0# proc.memory.rssMB‚ 3045.43359375) disk./.usagePercent‚75.77457998362377! disk.nvme1n1.out‚ 33153.546875 disk.md0.out‚ 670.44140625 disk.sda.in‚3.546875 disk.sda.out‚696.945800781250™-•2Ù- _wandbruntime‚6255  _runtime‚6255ÆÎ û'ÚýM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135447710 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚450441333612544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚435552378566496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚456567863020032 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚456567863020032 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _runtime‚6255.203455766 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135447710 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6125619422720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6103622006120 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚450441333612544 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚496998182434816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚435552361527136 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 247234614 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5349 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 135474080 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111786906 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6126529407488 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247260986 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚456566935032320 _step‚438 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6125619422720 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6402296025088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6102714030908 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚450441315609600 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99453322 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚503400229945344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 247234614 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚99453954 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6126529407488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚441656000572616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 247234614 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5349 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 135474080 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111786906 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6126529407488 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247260986 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚456566935032320 _step‚438 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6125619422720 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6402296025088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6102714030908 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚450441315609600 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99453322 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚503400229945344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 247234614 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚99453954 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6126529407488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚441656000572616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚450441333612544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚435552378566496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚456567863020032 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚456567863020032 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _runtime‚6255.203455766 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135447710 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6125619422720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6103622006120 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚450441333612544 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚496998182434816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚435552361527136 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚460621236202496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100329563 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6459768217600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 239267840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 249468871 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112774746 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6180585387520 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚630 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263207936 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDen·%/viceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚460622164176896 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112774744 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚454440650814976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚439424223659792 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 % _timestamp‚1.7885022538637488e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚460622164176896 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚439424240699152 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67567674 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32761259 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6180585387520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6459528949760 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6158381987742 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚501419218012672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚445582622686894 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 100328931 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚5399 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6157474012738 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚454440668817920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚507879010170368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136667781 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 _runtime‚6311.641086105 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 136694125 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136694125 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚460621236202496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100329563 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6459768217600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 239267840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 249468871 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112774746 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6180585387520 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚630 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263207936 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚460622164176896 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112774744 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚454440650814976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚439424223659792 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 % _timestamp‚1.7885022538637488e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚460622164176896 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚439424240699152 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67567674 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32761259 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6180585387520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6459528949760 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6158381987742 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚501419218012672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚445582622686894 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 100328931 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚5399 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6157474012738 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚454440668817920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚507879010170368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360¹é_=2ë- _wandbruntime‚6312  _runtime‚6312ËHg 'ìþM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 136692612 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 trainer/global_step‚5399 _runtime‚ 6312.64082482 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6181706847744 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚460703976748544 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚454522269900800 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136718982 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚501509467657216 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚439503241254336 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112794904 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100346727 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112794906 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136692612 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚454522287903744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32767620 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6460704023552 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 100347359 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249513888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67579739 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 249513888 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚454522269900800 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6159499261228 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6181706847744 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6158591286016 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6460926500864 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 249487516 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32766990 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚507970394158080 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚507970147740672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚445661832540352 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112794906 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚445662757554924 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 $ _timestamp‚1.788502254862784e+09 _step‚442 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚460704904736256 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚454522287903744 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249513888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67579739 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 249513888 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚454522269900800 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6159499261228 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6181706847744 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6158591286016 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6460926500864 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 249487516 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32766990 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚507970394158080 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚507970147740672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚445661832540352 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112794906 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚445662757554924 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 $ _timestamp‚1.788502254862784e+09 _step‚442 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚460704904736256 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚454522287903744 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 trainer/global_step‚5399 _runtime‚ 6312.64082482 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.s¦—-&’mall_pool.peak‚1231 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6181706847744 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚460703976748544 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚454522269900800 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136718982 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚501509467657216 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚439503241254336 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112794904 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100346727 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112794906 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136692612 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚454522287903744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32767620 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6460704023552 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615ºuÆ©â2î- _wandbruntime‚6312  _runtime‚6312ÄР@[ïÕ 2loss/SRqqt4/assignment_loss‚1.4953288473987154 .loss/FRt4/detection_loss‚2.932687977949778 /loss/FBt2/detection_loss‚2.4210184812545776 epoch‚4 _runtime‚6312.641814914 _step‚443 2loss/SRqqt3/assignment_loss‚1.0674168697951245 1loss/FBt3/assignment_loss‚0.19899800864770137 /loss/FRt2/detection_loss‚2.2677771231159567 1loss/SRqqt1/detection_loss‚1.9754603334835596 /loss/FBt4/detection_loss‚2.7420052029192448 1loss/FBt4/assignment_loss‚0.29861041588725357 .loss/FRt1/detection_loss‚1.581759770079294 /loss/FRt3/detection_loss‚1.6438179687266388 /loss/FBt3/detection_loss‚1.9237686031394534 %loss/total_loss‚40.75917275308118 $ _timestamp‚1.788502254864787e+09 0loss/FRt3/assignment_loss‚1.5872936441052345 /loss/FRt4/assignment_loss‚2.589672025044759 2loss/SRqqt2/assignment_loss‚1.6610133349895477 0loss/SRqqt4/detection_loss‚2.451730101449149 /loss/FBt1/detection_loss‚0.6464107057302079 0loss/FRt1/assignment_loss‚2.2906336131550016 1loss/SRqqt1/assignment_loss‚1.793783793066229 1loss/FBt1/assignment_loss‚0.18622750477304498 /loss/FRt2/assignment_loss‚3.166401943564415 trainer/global_step‚5399 0loss/FBt2/assignment_loss‚0.1439226248968715 1loss/SRqqt2/detection_loss‚2.8223856538534164 1loss/SRqqt3/detection_loss‚0.8710482060560025Kb¬8`ðÚ /loss/FBt4/detection_loss‚2.7420052029192448 1loss/FBt4/assignment_loss‚0.29861041588725357 .loss/FRt1/detection_loss‚1.581759770079294 /loss/FRt3/detection_loss‚1.6438179687266388 /loss/FBt3/detection_loss‚1.9237686031394534 %loss/total_loss‚40.75917275308118 $ _timestamp‚1.788502254864787e+09 0loss/FRt3/assignment_loss‚1.5872936441052345 /loss/FRt4/assignment_loss‚2.589672025044759 2loss/SRqqt2/assignment_loss‚1.6610133349895477 0loss/SRqqt4/detection_loss‚2.451730101449149 /loss/FBt1/detection_loss‚0.6464107057302079 0loss/FRt1/assignment_loss‚2.2906336131550016 1loss/SRqqt1/assignment_loss‚1.793783793066229 1loss/FBt1/assignment_loss‚0.18622750477304498 /loss/FRt2/assignment_loss‚3.166401943564415 trainer/global_step‚5399 0loss/FBt2/assignment_loss‚0.1439226248968715 1loss/SRqqt2/detection_loss‚2.8223856538534164 1loss/SRqqt3/detection_loss‚0.8710482060560025 2loss/SRqqt4/assignment_loss‚1.4953288473987154 .loss/FRt4/detection_loss‚2.932687977949778 /loss/FBt2/detection_loss‚2.4210184812545776 epoch‚4 _runtime‚6312.641814914 _step‚443 2loss/SRqqt3/assignment_loss‚1.0674168697951245 1loss/FBt3/assignment_loss‚0.19899800864770137 /loss/FRt2/detection_loss‚2.2677771231159567 1loss/SRqqt1/detection_loss‚1.9754603334835596»:î $‘ñ‚(:† ñÁéÔøý­k gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.916' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.229 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-jM_`ò‚(:Õ ñÁéÔùæ¬n% disk./.usageGB‚2710.6392250061035 disk.sdb2.out‚703.3857421875 disk.sda.in‚3.546875 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 676.6328125 network.sent‚ 1.3413533e+07 disk.rbd2.in‚0 disk.dm-0.out‚ 93616.859375 disk.nvme1n1.in‚ 743.53125 disk.rbd2.out‚ 1.908203125* proc.memory.percent‚18.587851524353027 disk.sdb1.in‚0 disk.md0p1.in‚ 4.1796875 disk.md0p1.out‚ 676.6328125 disk.dm-0.in‚ 1683.0703125 disk.sda2.in‚3.546875 disk.sda2.out‚703.3857421875 disk.sda1.in‚0 disk.rbd0.out‚ 1.68359375 disk.sdb.in‚ 0.6328125# proc.memory.rssMB‚ 3045.43359375 cpu‚25.16075155074013 network.recv‚ 2.080757e+06( disk./.usagePercent‚75.7906072218225 disk.sda1.out‚0 disk.sdb.out‚703.3857421875) proc.memory.availableMB‚ 1313.62890625 proc.cpu.threads‚17 disk.sda.out‚703.3857421875 disk.nvme0n1.in‚ 939.5390625 disk.nvme0n1.out‚ 60182.0625 disk.md0.in‚ 4.1796875$ memory_percent‚91.98224544525146 disk.sdb2.in‚ 0.6328125! disk.nvme1n1.out‚ 33434.796875õ"Úæ‘ó‚(:† €ÂéÔÒúÐk gpu.0.gpu‚10.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.61' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•¤Ðàeô‚(:Ú €ÂéÔÞ°¡m disk.sdb1.out‚0 disk.dm-0.in‚ 1683.0703125# disk.nvme0n1.out‚60370.16796875 network.recv‚ 2.086469e+06 disk.sda1.out‚0 disk.md0.in‚ 4.1796875$ memory_percent‚91.98248386383057 disk.rbd0.in‚0 disk.nvme0n1.in‚ 939.5390625( disk./.usagePercent‚75.7906072218225% disk./.usageGB‚2710.6392250061035 disk.sdb.out‚705.2841796875 network.sent‚ 1.3508583e+07 disk.rbd2.out‚ 1.908203125 disk.rbd0.out‚ 1.68359375 disk.dm-0.out‚93909.35546875 disk.sda2.out‚705.2841796875 disk.nvme1n1.out‚ 33539.1875 disk.md0.out‚ 678.4765625 cpu‚25.160459831670792 disk.sda.out‚705.2841796875 disk.md0p1.out‚ 678.4765625) proc.memory.availableMB‚ 1313.58984375 disk.sdb2.out‚705.2841796875 disk.sda.in‚3.546875 disk.sda1.in‚0 disk.md0p1.in‚ 4.1796875# proc.memory.rssMB‚ 3045.43359375 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.sdb.in‚ 0.6328125 disk.sda2.in‚3.546875 disk.nvme1n1.in‚ 743.53125* proc.memory.percent‚18.587851524353027 disk.sdb2.in‚ 0.6328125^n“õ‚(:ˆ ÂéÔíÉŸm gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚117.193' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.29825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Y=0Wö‚(:Ì ÂéÔÞÅÞm network.sent‚ 1.3514374e+07 disk.nvme1n1.in‚ 743.53125 disk.sdb1.in‚0 disk.sda2.in‚3.546875 disk.nvme0n1.in‚ 939.5390625 disk.rbd2.out‚ 1.908203125# memory_percent‚91.9809341430664# proc.memory.rssMB‚ 3045.43359375* proc.memory.percent‚18.587851524353027 proc.cpu.threads‚17 disk.md0p1.out‚ 679.95703125 disk.dm-0.in‚ 1683.0703125 disk.sda1.out‚0 cpu‚25.160156520095708 network.recv‚ 2.090391e+06 disk.sdb1.out‚0 disk.md0.in‚ 4.1796875 disk.sda1.in‚0) disk./.usagePercent‚75.79057874346043 disk.md0p1.in‚ 4.1796875 disk.dm-0.out‚ 94034.4453125 disk.rbd2.in‚0 disk.sdb2.out‚ 706.8359375 disk.sda.out‚ 706.8359375 disk.rbd0.out‚ 1.68359375 disk.sdb.out‚ 706.8359375 disk.sda2.out‚ 706.8359375 disk.sdb2.in‚ 0.6328125 disk.sda.in‚3.546875 disk.rbd0.in‚0 disk.sdb.in‚ 0.6328125 disk.nvme1n1.out‚ 33590.15625 disk.md0.out‚ 679.95703125% disk./.usageGB‚2710.6382064819336" disk.nvme0n1.out‚ 60444.2890625& proc.memory.availableMB‚ 1313.84375n6NO÷‚(:Ä žÂéÔÕÝÊl disk.sda1.out‚0) proc.memory.availableMB‚ 1313.59765625 cpu‚25.15986811709269 proc.cpu.threads‚17 disk.sdb2.out‚708.3125 disk.dm-0.out‚94347.48828125 network.sent‚ 1.3520685e+07 disk.sda.out‚708.3125 disk.md0p1.out‚ 681.3671875 disk.sdb.out‚708.3125" disk.nvme0n1.out‚ 60601.6328125$ memory_percent‚91.98243618011475* proc.memory.percent‚18.587851524353027 disk.sda2.in‚3.546875 disk.md0.out‚ 681.3671875 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 1.68359375 disk.sdb1.in‚0 disk.nvme1n1.in‚ 743.53125) disk./.usagePercent‚75.77582844515224 disk.sdb1.out‚0 disk.md0p1.in‚ 4.1796875# disk.nvme1n1.out‚33745.85546875 disk.md0.in‚ 4.1796875 disk.sdb.in‚ 0.6328125# proc.memory.rssMB‚ 3045.43359375 disk.sda2.out‚708.3125 disk.nvme0n1.in‚ 939.5390625 network.recv‚ 2.094809e+06 disk.rbd2.in‚0 disk.sdb2.in‚ 0.6328125 disk.sda.in‚3.546875 disk.dm-0.in‚ 1683.0703125$ disk./.usageGB‚2710.110664367676 disk.rbd2.out‚ 1.908203125Œ “ø‚(:ˆ žÂéÔûÜžm gpu.0.gpu‚20.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚177.231' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.30775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ‰zã2ù- _wandbruntime‚6369  _runtime‚6369€›ÆúÀ & lr-AdamW/pg2‚0.0007049259767644277 trainer/global_step‚5449 $ _timestamp‚1.788502311273688e+09 _runtime‚6369.050912049 _step‚444 & lr-AdamW/pg1‚0.0007049259767644277®zO,ËûÅ & lr-AdamW/pg1‚0.0007049259767644277 & lr-AdamW/pg2‚0.0007049259767644277 trainer/global_step‚5449 $ _timestamp‚1.788502311273688e+09 _runtime‚6369.050912049 _step‚444¼Ñ<2ü- _wandbruntime‚6369  _runtime‚6369‰ ü'ýöO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6214259242850 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6236672812544 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚443375120426352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6237582784000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 251695427 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885023112746503e+09 _runtime‚6369.051937174 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚458521605106176 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137939027 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 trainer/global_step‚5449 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚464759205893120 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚458521623109120 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 101222593 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚512448776567296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚594 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚512449022998016 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137912683 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137912683 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚505930527175168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 251721773 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33051671 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚464758277918720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚505930503235072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6213351267846 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113782744 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚464758277918720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚458521623109120 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113782744.ÈÐ_!þûO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137939027 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 trainer/global_step‚5449 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚464759205893120 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚458521623109120 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 101222593 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚512448776567296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚594 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚512449022998016 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137912683 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137912683 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚505930527175168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 251721773 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33051671 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚464758277918720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚505930503235072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6213351267846 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113782744 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚464758277918720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚458521623109120 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113782744 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6214259242850 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6236672812544 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚443375120426352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6237582784000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 251695427 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885023112746503e+09 _runtime‚6369.051937174 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚458521605106176½û€éq2ÿ- _wandbruntime‚6370  _runtime‚6370]¼^'€€N ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚443454138020896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68182989 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6519469044736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚68182987 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113802906 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚458603224192000 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 251740418 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚606 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137963884 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 trainer/global_step‚5449 _runtime‚6370.129817751 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33057433 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 251766790 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚458603242194944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚512540197984256 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 251766790 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚458603242194944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚449668589522660 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6238704257536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6238704257536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚464841018464768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233834496 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚464841946452480 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚608 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101240420 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚449669514537232 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚464841946452480 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137937514 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚458603224192000 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113802906 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚506020752879616 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6215376516336 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ÄÙ '…N L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 251766790 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚458603242194944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚449668589522660 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6238704257536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6238704257536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚464841018464768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233834496 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚464841946452480 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚608 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101240420 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚449669514537232 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚464841946452480 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137937514 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚458603224192000 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113802906 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚506020752879616 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6215376516336 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚443454138020896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68182989 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6519469044736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚68182987 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113802906 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚458603224192000 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 251740418 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚606 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137963884 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 trainer/global_step‚5449 _runtime‚6370.129817751 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33057433 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 251766790 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚458603242194944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚512540197984256 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚468895319634944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114790746 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batchÍ•÷<<"_start/reserved_bytes.small_pool.peak‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 139157585 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68774172 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6577016250880 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚462602577400320 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚510441812397568 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6269228522954 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102116557 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚510441788457472 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚468896247609344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚517019051139072 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚453595211637146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114790746 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚462602577400320 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139157585 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚453596136651510 _step‚449 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570‡GÖ(’ûO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114790746 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚462602577400320 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139157585 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚453596136651510 _step‚449 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚462602559397376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚6426.737954042 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 139183929 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚468895319634944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114790746 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 139157585 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68774172 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6577016250880 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚462602577400320 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚510441812397568 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6269228522954 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102116557 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚510441788457472 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚468896247609344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚517019051139072 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚453595211637146Á¯Ì·2“- _wandbruntime‚6427  _runtime‚6427 ƒ!'”€N H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚5499 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚462684196486144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚510532038102016 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚68786237 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6578187076096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 139182416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚453676271519540 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885023699042947e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚652 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚468978988168704 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚468978060180992 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚517110451849728 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚650 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚517110201238016 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6294791682560 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114810906 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114810904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6293881697792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6578413747712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6270345796232 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139208786 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33348172 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6293881697792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114810904 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚447405017748096 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254019692 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114810906 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚510532014161920 _step‚450 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚468978988168704 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102133759 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚462684196486144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33347522 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚6427.682218164 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 254019692 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139208786 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102134411 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68786239 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640«[ïIh•…N ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚462684196486144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33347522 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚6427.682218164 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 254019692 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139208786 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102134411 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68786239 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚5499 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚462684196486144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚510532038102016 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚68786237 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6578187076096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 139182416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚453676271519540 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885023699042947e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚652 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚468978988168704 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚468978060180992 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚517110451849728 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚650 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚517110201238016 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 YGDeviceStatsMon‰ˆ°£itor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6294791682560 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114810906 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114810904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6293881697792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6578413747712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6270345796232 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139208786 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33348172 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6293881697792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114810904 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚447405017748096 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254019692 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114810906 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚510532014161920 _step‚450 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚468978988168704 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102133759 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231Â3ƒgh2–- _wandbruntime‚6427  _runtime‚6427+«U÷Y—Ó /loss/FRt1/assignment_loss‚2.594827125072479 /loss/FRt3/assignment_loss‚1.611181657705734 2loss/SRqqt1/assignment_loss‚1.1827081804736876 2loss/SRqqt2/assignment_loss‚2.1150185376405717 2loss/SRqqt4/assignment_loss‚1.7235143532355626 1loss/FBt3/assignment_loss‚0.13071721203492156 1loss/FBt4/assignment_loss‚0.24753406840109166 /loss/FRt1/detection_loss‚1.7627647168468683 2loss/SRqqt3/assignment_loss‚1.0930246376877422 .loss/FRt2/detection_loss‚2.624614591851379 /loss/FRt4/detection_loss‚2.7102759369584017 1loss/SRqqt1/detection_loss‚2.1593576004871955 1loss/SRqqt3/detection_loss‚0.9854861586275747 trainer/global_step‚5499 .loss/FBt3/detection_loss‚2.288252362183162 .loss/FBt4/detection_loss‚2.202856035530567 1loss/SRqqt2/detection_loss‚2.6365171829859415 epoch‚4 $ _timestamp‚1.788502369906805e+09 _runtime‚6427.683838971 _step‚451 1loss/FBt2/assignment_loss‚0.10967627283377321 .loss/FRt3/detection_loss‚1.797533928235965 .loss/FRt2/assignment_loss‚3.91685849608797 /loss/FBt1/detection_loss‚0.8690741007769669 .loss/FBt2/detection_loss‚2.621140770117442 &loss/total_loss‚43.578531667689404 0loss/FRt4/assignment_loss‚3.2435989102651908 1loss/FBt1/assignment_loss‚0.20445256281520482 0loss/SRqqt4/detection_loss‚2.747546268834008 …ë­^˜Ø .loss/FRt2/assignment_loss‚3.91685849608797 /loss/FBt1/detection_loss‚0.8690741007769669 .loss/FBt2/detection_loss‚2.621140770117442 &loss/total_loss‚43.578531667689404 0loss/FRt4/assignment_loss‚3.2435989102651908 1loss/FBt1/assignment_loss‚0.20445256281520482 0loss/SRqqt4/detection_loss‚2.747546268834008 /loss/FRt1/assignment_loss‚2.594827125072479 /loss/FRt3/assignment_loss‚1.611181657705734 2loss/SRqqt1/assignment_loss‚1.1827081804736876 2loss/SRqqt2/assignment_loss‚2.1150185376405717 2loss/SRqqt4/assignment_loss‚1.7235143532355626 1loss/FBt3/assignment_loss‚0.13071721203492156 1loss/FBt4/assignment_loss‚0.24753406840109166 /loss/FRt1/detection_loss‚1.7627647168468683 2loss/SRqqt3/assignment_loss‚1.0930246376877422 .loss/FRt2/detection_loss‚2.624614591851379 /loss/FRt4/detection_loss‚2.7102759369584017 1loss/SRqqt1/detection_loss‚2.1593576004871955 1loss/SRqqt3/detection_loss‚0.9854861586275747 trainer/global_step‚5499 .loss/FBt3/detection_loss‚2.288252362183162 .loss/FBt4/detection_loss‚2.202856035530567 1loss/SRqqt2/detection_loss‚2.6365171829859415 epoch‚4 $ _timestamp‚1.788502369906805e+09 _runtime‚6427.683838971 _step‚451 1loss/FBt2/assignment_loss‚0.10967627283377321 .loss/FRt3/detection_loss‚1.797533928235965Ã1–)’™‚(:‡ éÂéÔÎõÖl gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚156.911' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.22775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Tk^¬_š‚(:Ô éÂéÔ¥ßp disk.rbd0.out‚ 1.68359375" disk.nvme1n1.out‚ 33747.5078125 disk.md0.in‚ 4.1796875 disk.sdb2.out‚ 716.68359375 disk.sdb1.out‚0 disk.md0p1.in‚ 4.1796875 disk.sda2.out‚ 716.68359375 disk.nvme0n1.in‚ 939.5390625) disk./.usagePercent‚75.78383342474851 disk.dm-0.in‚ 1683.0703125 disk.sdb1.in‚0 disk.sda2.in‚3.546875 disk.nvme1n1.in‚ 743.53125# proc.memory.rssMB‚ 3045.43359375 disk.sda.out‚ 716.68359375 disk.dm-0.out‚95365.59765625# disk.nvme0n1.out‚61618.08984375' proc.memory.availableMB‚ 1313.578125* proc.memory.percent‚18.587851524353027 network.recv‚ 2.115783e+06 disk.rbd2.out‚ 1.908203125 disk.sda1.out‚0 disk.md0p1.out‚ 689.43359375 disk.sdb.in‚ 0.6328125 cpu‚25.158326524902595 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.sdb.out‚ 716.68359375 proc.cpu.threads‚17 network.sent‚ 1.3663843e+07 disk.md0.out‚ 689.43359375 disk.sdb2.in‚ 0.6328125# memory_percent‚91.9825553894043$ disk./.usageGB‚2710.396961212158 disk.sda.in‚3.546875 disk.sda1.in‚0W‘n“›‚(:ˆ øÂéÔí±ìm gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.903' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.72575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸWI›aœ‚(:Ö øÂéÔ™ûƒo# disk.nvme1n1.out‚33747.66015625 disk.sda1.in‚0 disk.rbd0.in‚0 disk.md0p1.out‚ 690.87890625 disk.sda2.in‚3.546875 disk.nvme1n1.in‚ 743.53125 disk.sdb2.in‚ 0.6328125 disk.sda.in‚3.546875 disk.sda2.out‚ 718.193359375 disk.sdb.out‚ 718.193359375 network.recv‚ 2.121683e+06 disk.nvme0n1.in‚ 939.5390625 disk.md0.out‚ 690.87890625 disk.sda.out‚ 718.193359375 disk.md0p1.in‚ 4.1796875 disk.sdb.in‚ 0.6328125 proc.cpu.threads‚17# proc.memory.rssMB‚ 3045.43359375 disk.sda1.out‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 cpu‚25.15797363331725) disk./.usagePercent‚75.78383342474851$ memory_percent‚91.98117256164551 network.sent‚ 1.376079e+07 disk.nvme0n1.out‚ 61834.09375 disk.md0.in‚ 4.1796875 disk.rbd2.out‚ 1.908203125 disk.sdb2.out‚ 718.193359375 disk.rbd0.out‚ 1.68359375$ disk./.usageGB‚2710.396961212158 disk.rbd2.in‚0 disk.dm-0.in‚ 1683.0703125 disk.dm-0.out‚95581.75390625( proc.memory.availableMB‚ 1313.8046875* proc.memory.percent‚18.587851524353027ÂI3“‚(:ˆ ‡ÃéÔõªm gpu.0.gpu‚19.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚180.473' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.11825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0C‡îPž‚(:Å ‡ÃéÔá‘n disk.nvme1n1.in‚ 743.53125 network.recv‚ 2.125788e+06) disk./.usagePercent‚75.79082011624462 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 1683.0703125# disk.nvme1n1.out‚33747.75390625 disk.sda.out‚ 719.84375 disk.rbd0.in‚0 disk.sdb.in‚ 0.6328125 disk.nvme0n1.in‚ 939.5390625 disk.md0.out‚ 692.4609375% proc.memory.availableMB‚ 1313.4375# proc.memory.rssMB‚ 3045.43359375 cpu‚25.157692916323075 disk.sdb2.in‚ 0.6328125 disk.md0p1.out‚ 692.4609375 network.sent‚ 1.3764433e+07% disk./.usageGB‚2710.6468391418457 disk.md0.in‚ 4.1796875 disk.rbd2.out‚ 1.908203125 disk.sdb2.out‚ 719.84375 disk.md0p1.in‚ 4.1796875 disk.sdb.out‚ 719.84375$ memory_percent‚91.98341369628906 disk.sda1.out‚0 disk.sda2.in‚3.546875 disk.sda1.in‚0 disk.rbd0.out‚ 1.68359375* proc.memory.percent‚18.587851524353027 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sda.in‚3.546875 disk.dm-0.out‚95791.20703125 disk.sda2.out‚ 719.84375! disk.nvme0n1.out‚ 62043.453125»& “Ÿ‚(:ˆ –ÃéÔà›Ék gpu.0.gpu‚17.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚178.629' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.65725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éÚ _ ‚(:Ô –ÃéÔקÑl cpu‚25.15741826240356 disk.rbd0.in‚0 disk.md0p1.in‚ 4.1796875 disk.dm-0.in‚ 1683.0703125 disk.dm-0.out‚ 95827.96875 disk.sdb.out‚ 721.357421875 proc.cpu.threads‚17) disk./.usagePercent‚75.79082075620781 disk.sda.out‚ 721.357421875 disk.rbd0.out‚ 1.68359375 network.recv‚ 2.129952e+06 disk.sdb2.in‚ 0.6328125* proc.memory.percent‚18.587851524353027% disk./.usageGB‚2710.6468620300293" disk.nvme0n1.out‚ 62079.7421875 disk.nvme1n1.in‚ 743.53125 disk.md0.out‚ 693.9140625 disk.sdb2.out‚ 721.357421875$ memory_percent‚91.98489189147949( proc.memory.availableMB‚ 1313.1953125# proc.memory.rssMB‚ 3045.43359375 disk.nvme0n1.in‚ 939.5390625 disk.rbd2.out‚ 1.908203125 disk.sda1.out‚0 disk.md0p1.out‚ 693.9140625 network.sent‚ 1.3769887e+07 disk.sda2.out‚ 721.357421875" disk.nvme1n1.out‚ 33748.2265625 disk.sda.in‚3.546875 disk.sda1.in‚0 disk.sdb.in‚ 0.6328125 disk.md0.in‚ 4.1796875 disk.sdb1.out‚0 disk.sda2.in‚3.546875 disk.rbd2.in‚0 disk.sdb1.in‚0ÿÜÅ 2¡- _wandbruntime‚6484  _runtime‚64848R! Ç¢Á _step‚452 & lr-AdamW/pg1‚0.0006905650150263145 & lr-AdamW/pg2‚0.0006905650150263145 trainer/global_step‚5549 % _timestamp‚1.7885024267397733e+09 _runtime‚6484.517197827„Ì£Æ _step‚452 & lr-AdamW/pg1‚0.0006905650150263145 & lr-AdamW/pg2‚0.0006905650150263145 trainer/global_step‚5549 % _timestamp‚1.7885024267397733e+09 _runtime‚6484.517197827ÄcEnÒ2¤- _wandbruntime‚6484  _runtime‚6484,?u@ü'¥öO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 140428831 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚457602893633818 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 trainer/global_step‚5549 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6349757634048 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚473033289325568 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33634246 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚473032361351168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _runtime‚6484.518570193 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115798746 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚473033289325568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚451276862841392 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6325105778062 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 256227577 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6635745845760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚521588819525632 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6326013753066 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103011670 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚473032361351168 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 103011042 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69377424 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 _step‚453 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚466683531691520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7885024267408895e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6348847662592 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚466683513688576 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6348847662592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚451276879880752 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 256201231 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚466683531691520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6349757634048 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140402487 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140428831 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115798746 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115798744 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103011670 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚473032361351168 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 103011042 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69377424 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 _step‚453 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚466683531691520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7885024267408895e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6348847662592 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚466683513688576 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6348847662592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚451276879880752 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 256201231 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚466683531691520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6349757634048 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140402487 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140428831 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115798746 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.smalÔ æm¸ l_pool.current‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115798744 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚457602893633818 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 trainer/global_step‚5549 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6349757634048 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚473033289325568 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33634246 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚473032361351168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _runtime‚6484.518570193 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115798746 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368ÅÇ9—Š2§- _wandbruntime‚6485  _runtime‚6485~VC'¨€N I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚515043299384320 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚515043323324416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚473116029884928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140427318 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6350879107584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚466765132774400 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6349969122816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚521680231123456 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6350879107584 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 256272594 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 256272594 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103028891 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6637150022144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69389489 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 115818904 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚451355897475296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚451355880435936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6349969122816 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 256246222 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115818906 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115818904 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223104 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚521680473346560 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 _step‚454 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚473116029884928 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 % _timestamp‚1.7885024277821763e+09 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140427318 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6350879107584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚466765132774400 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6349969122816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚521680231123456 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6350879107584 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 256272594 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 256272594 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103028891 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6637150022144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69389489 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 115818904 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚451355897475296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚451355880435936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6349969122816 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 256246222 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115818906 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115818904 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223104 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚521680473346560 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 _step‚454 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚473116029884928 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 % _timestamp‚1.7885024277821763e+09 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640ÆEÍÛô2ª- _wandbruntime‚6485  _runtime‚6485µŸ'SY«Ó /loss/FRt2/detection_loss‚2.2180791541057476 /loss/FRt3/assignment_loss‚1.868774051704104 1loss/SRqqt2/assignment_loss‚1.092115505039692 1loss/SRqqt3/assignment_loss‚1.077171120240148 /loss/FBt1/detection_loss‚0.8368637245995078 /loss/FBt2/detection_loss‚2.3368822314909528 .loss/FBt3/detection_loss‚3.552192658185959 0loss/FRt4/assignment_loss‚3.1048796216646832 1loss/FBt1/assignment_loss‚0.12109382220233149 /loss/FBt4/detection_loss‚2.7206102079815335 epoch‚4 $loss/total_loss‚44.5988039536536 trainer/global_step‚5549 0loss/FBt3/assignment_loss‚0.7294017554668244 2loss/SRqqt1/assignment_loss‚1.4519983008503914 1loss/SRqqt4/detection_loss‚2.5984762559334436 _runtime‚6485.561623342 .loss/FRt4/detection_loss‚2.924458086490631 1loss/SRqqt3/detection_loss‚1.1298511843433825 _step‚455 2loss/SRqqt4/assignment_loss‚2.2291066149870553 0loss/FBt2/assignment_loss‚0.2740972781669565 /loss/FRt3/detection_loss‚1.7239056850473087 0loss/FBt4/assignment_loss‚0.1518118152733526 % _timestamp‚1.7885024277846265e+09 /loss/FRt1/detection_loss‚1.7861030094659833 0loss/FRt1/assignment_loss‚2.6167296767234802 /loss/FRt2/assignment_loss‚3.046047238742604 1loss/SRqqt1/detection_loss‚1.9552348935976624 1loss/SRqqt2/detection_loss‚3.0529200613498686†µV‚^¬Ø $loss/total_loss‚44.5988039536536 trainer/global_step‚5549 0loss/FBt3/assignment_loss‚0.7294017554668244 2loss/SRqqt1/assignment_loss‚1.4519983008503914 1loss/SRqqt4/detection_loss‚2.5984762559334436 _runtime‚6485.561623342 .loss/FRt4/detection_loss‚2.924458086490631 1loss/SRqqt3/detection_loss‚1.1298511843433825 _step‚455 2loss/SRqqt4/assignment_loss‚2.2291066149870553 0loss/FBt2/assignment_loss‚0.2740972781669565 /loss/FRt3/detection_loss‚1.7239056850473087 0loss/FBt4/assignment_loss‚0.1518118152733526 % _timestamp‚1.7885024277846265e+09 /loss/FRt1/detection_loss‚1.7861030094659833 0loss/FRt1/assignment_loss‚2.6167296767234802 /loss/FRt2/assignment_loss‚3.046047238742604 1loss/SRqqt1/detection_loss‚1.9552348935976624 1loss/SRqqt2/detection_loss‚3.0529200613498686 /loss/FRt2/detection_loss‚2.2180791541057476 /loss/FRt3/assignment_loss‚1.868774051704104 1loss/SRqqt2/assignment_loss‚1.092115505039692 1loss/SRqqt3/assignment_loss‚1.077171120240148 /loss/FBt1/detection_loss‚0.8368637245995078 /loss/FBt2/detection_loss‚2.3368822314909528 .loss/FBt3/detection_loss‚3.552192658185959 0loss/FRt4/assignment_loss‚3.1048796216646832 1loss/FBt1/assignment_loss‚0.12109382220233149 /loss/FBt4/detection_loss‚2.7206102079815335 epoch‚4ÇH¡Ó“­‚(:ˆ ¥ÃéÔ„°m gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚161.587' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.39675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%¬ío®‚(:ä ¥ÃéÔƒ©÷n disk.md0.out‚ 695.76953125 disk.sdb2.in‚ 0.6328125 disk.rbd0.in‚0 disk.md0p1.out‚ 695.76953125) proc.memory.availableMB‚ 1313.64453125* proc.memory.percent‚18.587851524353027! disk.sdb2.out‚723.27783203125! disk.sda2.out‚723.27783203125 disk.rbd0.out‚ 1.68359375 disk.rbd2.out‚ 1.908203125 disk.sda.out‚723.27783203125 disk.sda1.in‚0 disk.dm-0.in‚ 1683.0703125# proc.memory.rssMB‚ 3045.43359375# disk.nvme0n1.out‚62540.75390625 network.sent‚ 1.3787957e+07) disk./.usagePercent‚75.79083440875593 disk.rbd2.in‚0 disk.sdb.out‚723.27783203125$ memory_percent‚91.98215007781982 proc.cpu.threads‚17 network.recv‚ 2.133339e+06 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.out‚96289.12109375" disk.nvme1n1.out‚ 33748.3671875% disk./.usageGB‚2710.6473503112793 disk.md0.in‚ 4.1796875 disk.sda.in‚3.546875 disk.md0p1.in‚ 4.1796875 disk.sdb.in‚ 0.6328125 disk.sda1.out‚0 disk.nvme0n1.in‚ 939.5390625 disk.nvme1n1.in‚ 743.53125 cpu‚25.157075890823617 disk.sda2.in‚3.546875tS‚(:ˆ ´ÃéԻƀm gpu.0.gpu‚20.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚178.297' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.57425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„”—}g°‚(:Ü ´ÃéÔÊÉám$ disk./.usageGB‚2710.647331237793 disk.nvme0n1.in‚ 939.5390625 disk.sda.out‚724.76025390625' proc.memory.availableMB‚ 1313.640625 cpu‚25.156801325057824! disk.sda2.out‚724.76025390625! disk.sdb2.out‚724.76025390625 disk.sdb1.in‚0 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd2.out‚ 1.908203125 disk.rbd0.out‚ 1.68359375* proc.memory.percent‚18.587851524353027 network.sent‚ 1.3885149e+07 network.recv‚ 2.138855e+06 disk.sda2.in‚3.546875 disk.md0.out‚ 697.1953125 disk.sda1.in‚0 disk.md0p1.in‚ 4.1796875# proc.memory.rssMB‚ 3045.43359375) disk./.usagePercent‚75.79083387545327" disk.nvme0n1.out‚ 62785.9765625! disk.nvme1n1.out‚ 33748.546875 disk.sda.in‚3.546875 disk.sdb.in‚ 0.6328125 disk.nvme1n1.in‚ 743.53125 disk.md0.in‚ 4.1796875 disk.sdb2.in‚ 0.6328125 disk.sda1.out‚0 disk.sdb1.out‚0 disk.dm-0.out‚ 96534.5234375$ memory_percent‚91.98217391967773 disk.sdb.out‚724.76025390625 disk.rbd0.in‚0 disk.md0p1.out‚ 697.1953125 disk.dm-0.in‚ 1683.0703125½„©±‚(:’ ÃÃéÔöþˆm gpu.0.gpu‚35.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚119.368' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.841999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BF¸j²‚(:ß ÃÃéÔ™¾¼m disk.sda1.in‚0 disk.md0p1.out‚ 698.54296875# proc.memory.rssMB‚ 3045.43359375* proc.memory.percent‚18.587851524353027 disk.nvme0n1.out‚ 62846.15625 disk.rbd2.in‚0 disk.sdb.out‚726.16357421875$ memory_percent‚91.98212623596191 disk.sda.out‚726.16357421875 disk.sda1.out‚0 disk.dm-0.out‚96594.83203125 disk.sdb2.in‚ 0.6328125 disk.rbd0.in‚0 disk.rbd0.out‚ 1.68359375 disk.sdb.in‚ 0.6328125 network.sent‚ 1.3891049e+07 disk.sda2.in‚3.546875! disk.sda2.out‚726.16357421875 disk.rbd2.out‚ 1.908203125( proc.memory.availableMB‚ 1313.6484375( disk./.usagePercent‚75.7746600856834$ disk./.usageGB‚2710.068878173828 disk.nvme1n1.in‚ 743.53125! disk.sdb2.out‚726.16357421875 disk.sda.in‚3.546875 disk.md0p1.in‚ 4.1796875# disk.nvme1n1.out‚33748.67578125 disk.nvme0n1.in‚ 939.5390625 disk.md0.out‚ 698.54296875 disk.sdb1.in‚0 disk.sdb1.out‚0 cpu‚25.156520868912764 proc.cpu.threads‚17 disk.dm-0.in‚ 1683.0703125 network.recv‚ 2.143019e+06 disk.md0.in‚ 4.1796875ÆI’³‚(:‡ ÒÃéÔÒÂm gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.562' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.6405 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çe´‚(:Ú ÒÃéÔ®øÖm disk.md0.in‚ 4.1796875) proc.memory.availableMB‚ 1313.39453125 network.recv‚ 2.147813e+06 disk.sda.in‚3.546875 disk.sdb1.in‚0 disk.dm-0.out‚ 96804.859375! disk.nvme1n1.out‚ 33748.890625 proc.cpu.threads‚17 disk.sda.out‚728.17626953125 disk.sda1.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 700.4765625# proc.memory.rssMB‚ 3045.43359375 network.sent‚ 1.3895166e+07 disk.nvme1n1.in‚ 743.53125 disk.rbd2.out‚ 1.908203125# disk./.usageGB‚2710.11661529541 disk.sdb.in‚ 0.6328125 disk.sdb.out‚728.17626953125 disk.nvme0n1.out‚ 63055.96875 cpu‚25.156237775035123) disk./.usagePercent‚75.77599483558237 disk.rbd0.in‚0$ memory_percent‚91.98367595672607 disk.rbd0.out‚ 1.68359375 disk.md0p1.out‚ 700.4765625 disk.sda2.in‚3.546875 disk.nvme0n1.in‚ 939.5390625 disk.rbd2.in‚0* proc.memory.percent‚18.587851524353027 disk.sdb2.in‚ 0.6328125! disk.sdb2.out‚728.17626953125 disk.sda1.out‚0! disk.sda2.out‚728.17626953125 disk.md0p1.in‚ 4.1796875 disk.dm-0.in‚ 1683.0703125ÛÂ%2µ- _wandbruntime‚6542  _runtime‚6542±¿®Ry¶Á _runtime‚6542.000115403 _step‚456 & lr-AdamW/pg1‚0.0006833323967282631 & lr-AdamW/pg2‚0.0006833323£=åN967282631 trainer/global_step‚5599 % _timestamp‚1.7885024842228146e+09– ªÌ·Æ & lr-AdamW/pg2‚0.0006833323967282631 trainer/global_step‚5599 % _timestamp‚1.7885024842228146e+09 _runtime‚6542.000115403 _step‚456 & lr-AdamW/pg1‚0.0006833323967282631ÈY6Ñ[2¸- _wandbruntime‚6542  _runtime‚6542Ëâ6Ùü'¹öO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6404935087616 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6694964513792 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103907046 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚455227759607952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69980674 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69980672 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚526159098828288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 141673733 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚470764485982720 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 258480479 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 103906417 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚519464358902272 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚457 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6405845059072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6381891008174 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6404935087616 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚477170331041792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 141673733 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33926372 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚526159347356160 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 % _timestamp‚1.7885024842240765e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6405845059072 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141647389 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚477169403067392 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116806744 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚477170331041792 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6694739926016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6380983033170 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116806746 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141647389 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚477169403067392 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚470764467979776 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚519464382842368 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚5599 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚461609650616126 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚470764485982720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚6542.001418829 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚461608725601762 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570²øýÈ(ºûO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6404935087616 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚477170331041792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 141673733 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33926372 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚526159347356160 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 % _timestamp‚1.7885024842240765e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6405845059072 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141647389 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚477169403067392 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116806744 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚477170331041792 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6694739926016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6380983033170 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116806746 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141647389 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚477169403067392 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚470764467979776 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚519464382842368 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚5599 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚461609650616126 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚470764485982720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚6542.001418829 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚461608725601762 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6404935087616 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6694964513792 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103907046 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚455227759607952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69980674 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69980672 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚526159098828288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 141673733 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚470764485982720 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 258480479 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 103906417 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚519464358902272 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚457 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6405845059072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6381891008174ÉýJ(2»- _wandbruntime‚6543  _runtime‚6543±ðÍ/'¼€N QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6406966532608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚627 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚455306760163136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6406056547840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33932194 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69992737 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚526250749437952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6695912122368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚461688860469584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6406966532608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚519554608546816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103924304 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116826904 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚519554584606720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚455306777202496 _runtime‚6543.044538919 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6696140891136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 % _timestamp‚1.7885024852658925e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚470846087065600 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚629 _step‚458 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚477252143613440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 258499124 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6406056547840 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6383008281660 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 trainer/global_step‚5599 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚477253071601152 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116826906 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116826904 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141698590 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚526250496729088 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33931567 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 258525496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864™!{EG½…N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚477253071601152 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 258525496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6406966532608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚627 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚455306760163136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6406056547840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33932194 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69992737 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚526250749437952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6695912122368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requeéÞ^èÄsted_bytes.all.freed‚461688860469584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6406966532608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚519554608546816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103924304 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116826904 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚519554584606720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚455306777202496 _runtime‚6543.044538919 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6696140891136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 % _timestamp‚1.7885024852658925e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚470846087065600 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚629 _step‚458 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚477252143613440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 258499124 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6406056547840 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6383008281660 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 trainer/global_step‚5599 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚477253071601152 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116826906 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116826904 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141698590 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚526250496729088 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33931567 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 trainer/global_step‚5649 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚459178622295792 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142892291 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70583922 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 260707035 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚465615482584070 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚622 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚474845422270976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6461022512640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚474845440273920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚523975668064768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚530729048202240 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6753609790976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚459178639335152 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117814744 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117814744 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚481306444783616 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6461932484096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 260733381 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚474845440273920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚523975668064768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚530729048202240 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6753609790976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚459178639335152 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117814744 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117814744 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚481306444783616 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6461932484096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 260733381 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 trainer/global_step‚5649 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚459178622295792 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142892291 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70583922 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 260707035 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚465615482584070ÍSN@“Í‚(:ˆ ŽÄéÔ÷߯n gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.637' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.15925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚ÝÙEg΂(:Ü ŽÄéÔÆ™Èn disk.rbd0.out‚ 1.68359375) disk./.usagePercent‚75.78036813072042# proc.memory.rssMB‚ 3045.43359375 disk.sda2.in‚3.546875" disk.nvme1n1.out‚ 33749.5546875 disk.md0.out‚ 707.16796875 disk.sdb2.in‚ 0.6328125 disk.rbd0.in‚0$ memory_percent‚91.98379516601562$ proc.memory.availableMB‚1313.375% disk./.usageGB‚2710.2730255126953 disk.md0p1.out‚ 707.16796875 disk.dm-0.out‚ 97443.046875 disk.sdb.out‚735.11767578125" disk.nvme0n1.out‚ 63693.4921875 disk.rbd2.out‚ 1.908203125 disk.sda.in‚3.546875 disk.md0p1.in‚ 4.1796875 disk.nvme1n1.in‚ 743.53125 disk.sda1.in‚0 disk.sdb.in‚ 0.6328125 disk.md0.in‚ 4.1796875 disk.sda.out‚735.11767578125 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.rbd2.in‚0* proc.memory.percent‚18.587851524353027 cpu‚25.155080029572392! disk.sdb2.out‚735.11767578125 disk.sdb1.in‚0 network.sent‚ 1.4020126e+07 network.recv‚ 2.165302e+06 disk.dm-0.in‚ 1683.0703125! disk.sda2.out‚735.11767578125 disk.nvme0n1.in‚ 939.5390625 disk.sda1.out‚0U+ U2Ï- _wandbruntime‚6600  _runtime‚6600î¼"Å'ЀN ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚481390113317376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6462143972864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70595989 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚607 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6754591211008 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚459257639890336 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6462143972864 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚474927041356800 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚459257656929696 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6438885536768 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142917122 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚474927059359744 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34223468 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6754811591168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260752026 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 260778398 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260778398 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117834904 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117834906 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚609 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚474927059359744 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104818848 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚465695617451892 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚524065893769216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117834904 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34222861 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 104819457 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70595987 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚6600.530117636 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117834906 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6463053957632 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 260752026 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚530820461040128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚465696542466464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚462 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220380160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚481390113317376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142943492 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244320256 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142943492 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 % _timestamp‚1.7885025427522058e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚474927041356800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚481389185329664 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚524065869829120 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6437977561556 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0§ð¦ 'Ñ…N WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚481390113317376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142943492 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244320256 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142943492 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 % _timestamp‚1.7885025427522058e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚474927041356800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚481389185329664 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚524065869829120 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6437977561556 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚481390113317376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6462143972864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70595989 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚607 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6754591211008 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚459257639890336 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6462143972864 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚474927041356800 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚459257656929696 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6438885536768 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142917122 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚474927059359744 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34223468 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6754811591168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260752026 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 260778398 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260778398 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117834904 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117834906 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚609 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚474927059359744 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104818848 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚465695617451892 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚524065893769216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117834904 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34222861 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 104819457 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70595987 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚6600.530117636 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117834906 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6463053957632 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 260752026 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚530820461040128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚465696542466464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚462 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220380160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640Ϊþ³2Ò- _wandbruntime‚6600  _runtime‚6600ÖÅy®ZÓÔ 2loss/SRqqt2/assignment_loss‚1.4493586549965234 0loss/FBt2/assignment_loss‚0.3510238718157173 1loss/SRqqt2/detection_loss‚2.3907811229045572 .loss/FBt4/detection_loss‚2.348558408021927 0loss/FRt4/assignment_loss‚3.0208986428650944 .loss/FRt1/detection_loss‚1.723204169898892 0loss/SRqqt4/detection_loss‚2.404186772949555 epoch‚4 trainer/global_step‚5649 0loss/SRqqt1/detection_loss‚2.186912110873631 %loss/total_loss‚41.90401030797938 $ _timestamp‚1.788502542754222e+09 /loss/FRt1/assignment_loss‚2.565576071832694 0loss/FRt3/assignment_loss‚1.8562304983878959 0loss/FBt3/assignment_loss‚0.7032978681441919 /loss/FRt2/detection_loss‚2.3330501971973314 /loss/FRt3/detection_loss‚1.9321512612032479 1loss/SRqqt3/detection_loss‚0.8945463510068852 /loss/FBt2/detection_loss‚1.9658126963509455 2loss/SRqqt1/assignment_loss‚1.7315290993345636 2loss/SRqqt3/assignment_loss‚0.7343657197812059 /loss/FBt3/detection_loss‚2.0075930489434137 1loss/SRqqt4/assignment_loss‚1.567911308916176 1loss/FBt1/assignment_loss‚0.17370925249693916 1loss/FBt4/assignment_loss‚0.15521473987804105 0loss/FRt2/assignment_loss‚3.0968597763114505 /loss/FRt4/detection_loss‚3.3171601078727027 _step‚463 /loss/FBt1/detection_loss‚0.9940785559957993 _runtime‚6600.531197929¯ò_ÔÙ 0loss/SRqqt1/detection_loss‚2.186912110873631 %loss/total_loss‚41.90401030797938 $ _timestamp‚1.788502542754222e+09 /loss/FRt1/assignment_loss‚2.565576071832694 0loss/FRt3/assignment_loss‚1.8562304983878959 0loss/FBt3/assignment_loss‚0.7032978681441919 /loss/FRt2/detection_loss‚2.3330501971973314 /loss/FRt3/detection_loss‚1.9321512612032479 1loss/SRqqt3/detection_loss‚0.8945463510068852 /loss/FBt2/detection_loss‚1.9658126963509455 2loss/SRqqt1/assignment_loss‚1.7315290993345636 2loss/SRqqt3/assignment_loss‚0.7343657197812059 /loss/FBt3/detection_loss‚2.0075930489434137 1loss/SRqqt4/assignment_loss‚1.567911308916176 1loss/FBt1/assignment_loss‚0.17370925249693916 1loss/FBt4/assignment_loss‚0.15521473987804105 0loss/FRt2/assignment_loss‚3.0968597763114505 /loss/FRt4/detection_loss‚3.3171601078727027 _step‚463 /loss/FBt1/detection_loss‚0.9940785559957993 _runtime‚6600.531197929 2loss/SRqqt2/assignment_loss‚1.4493586549965234 0loss/FBt2/assignment_loss‚0.3510238718157173 1loss/SRqqt2/detection_loss‚2.3907811229045572 .loss/FBt4/detection_loss‚2.348558408021927 0loss/FRt4/assignment_loss‚3.0208986428650944 .loss/FRt1/detection_loss‚1.723204169898892 0loss/SRqqt4/detection_loss‚2.404186772949555 epoch‚4 trainer/global_step‚5649Ïšâ‘°`Õ‚(:Õ ÄéÔ–ͼo disk.md0.in‚ 4.1796875 disk.rbd0.in‚0$ memory_percent‚91.98384284973145* proc.memory.percent‚18.587851524353027 cpu‚25.154835409745214 disk.md0p1.in‚ 4.1796875 disk.md0.out‚ 708.609375 disk.sda.in‚3.546875 network.sent‚ 1.4115635e+07 disk.dm-0.out‚97740.91796875 disk.sda2.in‚3.546875 disk.sda.out‚ 736.619140625 disk.rbd0.out‚ 1.68359375( proc.memory.availableMB‚ 1313.3671875 disk.sdb1.out‚0 disk.nvme1n1.in‚ 743.53125 network.recv‚ 2.169523e+06 disk.sdb1.in‚0 disk.dm-0.in‚ 1683.0703125 disk.sdb.in‚ 0.6328125 disk.rbd2.in‚0) disk./.usagePercent‚75.78386670283454 disk.sdb.out‚ 736.619140625 disk.rbd2.out‚ 1.908203125 disk.sdb2.in‚ 0.6328125 proc.cpu.threads‚17 disk.sda2.out‚ 736.619140625# disk.nvme0n1.out‚63991.12109375 disk.sda1.in‚0$ disk./.usageGB‚2710.398151397705 disk.md0p1.out‚ 708.609375 disk.sdb2.out‚ 736.619140625# proc.memory.rssMB‚ 3045.43359375 disk.sda1.out‚0 disk.nvme0n1.in‚ 939.5390625! disk.nvme1n1.out‚ 33749.7968758ôdG‘Ö‚(:† ÄéÔýÛ…o gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚175.512' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è¸Æˆ“ׂ(:ˆ ¬ÄéÔÈž˜m gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.129' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.03225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖŸxŽiØ‚(:Þ ¬ÄéÔì€Ñm# disk.nvme1n1.out‚33750.06640625* proc.memory.percent‚18.587851524353027) disk./.usagePercent‚75.79087088665791 disk.sdb1.out‚0 disk.sdb.in‚ 0.6328125! disk.sda2.out‚738.48486328125 disk.md0.in‚ 4.1796875' proc.memory.availableMB‚ 1313.359375 disk.rbd0.out‚ 1.68359375 disk.sdb1.in‚0 disk.sda.out‚738.48486328125 disk.md0p1.out‚ 710.4140625 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.md0p1.in‚ 4.1796875 network.sent‚ 1.4212852e+07$ disk./.usageGB‚2710.648654937744 disk.sda1.out‚0 disk.nvme1n1.in‚ 743.53125 disk.md0.out‚ 710.4140625$ memory_percent‚91.98389053344727 disk.sda.in‚3.546875 disk.sda1.in‚0 disk.dm-0.out‚98010.92578125 network.recv‚ 2.175169e+06 disk.dm-0.in‚ 1683.0703125 disk.sdb.out‚738.48486328125 disk.sda2.in‚3.546875 disk.nvme0n1.in‚ 939.5390625 disk.rbd2.out‚ 1.908203125# proc.memory.rssMB‚ 3045.43359375 cpu‚25.154538560160717 disk.sdb2.in‚ 0.6328125! disk.sdb2.out‚738.48486328125! disk.nvme0n1.out‚ 64260.859375ðÁzµ“Ù‚(:ˆ »ÄéÔ·¡Çl gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚159.419' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.85475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º=NÚ‚(:à »ÄéÔ•³Îm* proc.memory.percent‚18.587851524353027# disk.nvme0n1.out‚64347.35546875) disk./.usagePercent‚75.79084240829583 network.recv‚ 2.179143e+06 disk.sda.out‚739.95849609375 disk.sda1.in‚0 disk.md0p1.out‚ 711.8203125 disk.sdb.out‚739.95849609375 disk.rbd2.in‚0 disk.rbd2.out‚ 1.908203125$ memory_percent‚91.98403358459473 disk.sdb2.in‚ 0.6328125 disk.sdb1.out‚0 disk.sdb.in‚ 0.6328125( proc.memory.availableMB‚ 1313.3359375 cpu‚25.154266321882513 proc.cpu.threads‚17! disk.sdb2.out‚739.95849609375 disk.sda2.in‚3.546875 disk.md0.in‚ 4.1796875 disk.md0.out‚ 711.8203125 network.sent‚ 1.4220882e+07$ disk./.usageGB‚2710.647636413574 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.dm-0.in‚ 1683.0703125 disk.dm-0.out‚ 98097.5859375! disk.sda2.out‚739.95849609375# disk.nvme1n1.out‚33750.23046875 disk.sda.in‚3.546875 disk.sda1.out‚0 disk.rbd0.out‚ 1.68359375# proc.memory.rssMB‚ 3045.43¹ž$c359375 disk.md0p1.in‚ 4.1796875 disk.nvme0n1.in‚ 939.5390625 disk.nvme1n1.in‚ 743.53125÷52Û- _wandbruntime‚6657  _runtime‚6657ç"?gÇÜÁ & lr-AdamW/pg2‚0.0006687704212941233 trainer/global_step‚5699 % _timestamp‚1.7885025992948985e+09 _runtime‚6657.072121346 _step‚464 & lr-AdamW/pg1‚0.0006687704212941233—õØÝÌÝÆ % _timestamp‚1.7885025992948985e+09 _runtime‚6657.072121346 _step‚464 & lr-AdamW/pg1‚0.0006687704212941233 & lr-AdamW/pg2‚0.0006687704212941233 trainer/global_step‚5699Ðk˜žÍ2Þ- _wandbruntime‚6657  _runtime‚6657ûšËü'ßöO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118822744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚535299050744320 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144137193 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144137193 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118822746 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6517109937664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71187174 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6492737543386 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885025992959433e+09 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105696144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚535299305563648 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚469623164580742 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚463129502022992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚6657.073237001 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚478926376562176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚528486953287168 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 105696781 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚478926394565120 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6812121397248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚469622239566378 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118822746 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6518019909120 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚485444414474240 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262986283 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚485443486499840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚478926394565120 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚485443486499840 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚528486929347072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚478926376562176 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118822744 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34508972 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚465 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6493645518390 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71187172 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144163537 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6812352276480 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725Ùeêˆ(àûO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6517109937664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71187174 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6492737543386 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885025992959433e+09 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105696144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚535299305563648 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚469623164580742 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚463129502022992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚6657.073237001 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚478926376562176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚528486953287168 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 105696781 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚478926394565120 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6812121397248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚469622239566378 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118822746 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6518019909120 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚485444414474240 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262986283 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚485443486499840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚478926394565120 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚485443486499840 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚528486929347072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚478926376562176 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118822744 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34508972 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚465 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6493645518390 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71187172 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144163537 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6812352276480 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118822744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚535299050744320 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144137193 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144137193 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118822746 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0Ñø;‚è2á- _wandbruntime‚6658  _runtime‚6658ŸïI'â€N UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6518231397888 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚485526227045888 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 % _timestamp‚1.7885026004025574e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚485527155033600 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚463208519617536 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6493854816664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6519141382656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6813521070080 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144162024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚469702374434200 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚463208536656896 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 263031300 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚469703299448772 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚479008013650944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244320256 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263004928 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118842904 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚535390455741440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 105714496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34515257 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220380160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚466 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 144188394 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚528577178991616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105713876 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚479007995648000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263031300 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6494762791876 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263004928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚485527155033600 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚528577155051520 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144188394 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144162024 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚479008013650944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚535390700061696 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚479007995648000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6518231397888 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118842906 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118842904 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 trainer/global_step‚5699 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _runtime‚6658.180331992)Nˆá+ã…N ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚479007995648000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6518231397888 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118842906 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118842904 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 trainer/global_step‚5699 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _runtime‚6658.180331992 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6518231397888 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚485526227045888 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatŽ÷à sMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 % _timestamp‚1.7885026004025574e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚485527155033600 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚463208519617536 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6493854816664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6519141382656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6813521070080 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144162024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚469702374434200 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚463208536656896 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 263031300 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚469703299448772 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚479008013650944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244320256 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263004928 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118842904 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚535390455741440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 105714496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34515257 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220380160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚466 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 144188394 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚528577178991616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105713876 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚479007995648000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263031300 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6494762791876 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263004928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚485527155033600 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚528577155051520 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144188394 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144162024 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚479008013650944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚535390700061696 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570Òd¢)72ä- _wandbruntime‚6658  _runtime‚6658Z£ÿ_åÙ 1loss/FBt3/assignment_loss‚0.06174973990824952 1loss/FBt4/assignment_loss‚0.05092099710236653 1loss/SRqqt4/detection_loss‚2.2646867242352715 %loss/total_loss‚41.10589537119856 _step‚467 0loss/FRt1/assignment_loss‚2.6917222628467963 2loss/SRqqt3/assignment_loss‚1.2738350369628857 1loss/FBt2/assignment_loss‚0.45946972512410866 .loss/FRt2/detection_loss‚2.206363200365255 /loss/FRt4/detection_loss‚2.8021711861645735 /loss/FBt2/detection_loss‚2.4994101772705712 0loss/FRt4/assignment_loss‚3.2494878327405012 1loss/SRqqt2/assignment_loss‚1.391262294917271 2loss/SRqqt1/assignment_loss‚1.2408876167610288 1loss/SRqqt1/detection_loss‚1.8769535087049007 /loss/FBt3/detection_loss‚1.8808447957038879 trainer/global_step‚5699 _runtime‚6658.182339581 0loss/FRt3/assignment_loss‚1.7886533391648445 .loss/FRt1/detection_loss‚2.063646500799989 /loss/FBt4/detection_loss‚2.1970000161844143 epoch‚4 % _timestamp‚1.7885026004052997e+09 0loss/FRt2/assignment_loss‚3.1293028133610883 1loss/FBt1/assignment_loss‚0.13235794245522609 /loss/FRt3/detection_loss‚1.6774608560878297 0loss/SRqqt3/detection_loss‚1.008901339125904 /loss/FBt1/detection_loss‚0.8263352600495435 2loss/SRqqt4/assignment_loss‚1.9304594561971467 1loss/SRqqt2/detection_loss‚2.4020127489649017Ô¦u@dæÞ 0loss/FRt1/assignment_loss‚2.6917222628467963 2loss/SRqqt3/assignment_loss‚1.2738350369628857 1loss/FBt2/assignment_loss‚0.45946972512410866 .loss/FRt2/detection_loss‚2.206363200365255 /loss/FRt4/detection_loss‚2.8021711861645735 /loss/FBt2/detection_loss‚2.4994101772705712 0loss/FRt4/assignment_loss‚3.2494878327405012 1loss/SRqqt2/assignment_loss‚1.391262294917271 2loss/SRqqt1/assignment_loss‚1.2408876167610288 1loss/SRqqt1/detection_loss‚1.8769535087049007 /loss/FBt3/detection_loss‚1.8808447957038879 trainer/global_step‚5699 _runtime‚6658.182339581 0loss/FRt3/assignment_loss‚1.7886533391648445 .loss/FRt1/detection_loss‚2.063646500799989 /loss/FBt4/detection_loss‚2.1970000161844143 epoch‚4 % _timestamp‚1.7885026004052997e+09 0loss/FRt2/assignment_loss‚3.1293028133610883 1loss/FBt1/assignment_loss‚0.13235794245522609 /loss/FRt3/detection_loss‚1.6774608560878297 0loss/SRqqt3/detection_loss‚1.008901339125904 /loss/FBt1/detection_loss‚0.8263352600495435 2loss/SRqqt4/assignment_loss‚1.9304594561971467 1loss/SRqqt2/detection_loss‚2.4020127489649017 1loss/FBt3/assignment_loss‚0.06174973990824952 1loss/FBt4/assignment_loss‚0.05092099710236653 1loss/SRqqt4/detection_loss‚2.2646867242352715 %loss/total_loss‚41.10589537119856 _step‚467Ó"MU‘ç‚(:† ÊÄéÔ¯Ýál gpu.0.gpu‚32.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.61' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èuhè‚(:Ý ÊÄéÔ®Ÿ§n disk.rbd0.out‚ 1.68359375 network.sent‚ 1.4237901e+07 disk.md0.in‚ 4.1796875* proc.memory.percent‚18.587851524353027 disk.md0.out‚ 713.421875' proc.memory.availableMB‚ 1312.828125 cpu‚25.153989356551577 disk.md0p1.in‚ 4.1796875 disk.md0p1.out‚ 713.421875! disk.sda2.out‚741.62548828125 disk.sdb2.in‚ 0.6328125! disk.sdb2.out‚741.62548828125 disk.sda1.out‚0 disk.dm-0.in‚ 1683.0703125# proc.memory.rssMB‚ 3045.43359375) disk./.usagePercent‚75.79084240829583 disk.sdb.in‚ 0.6328125 disk.sda2.in‚3.546875# disk.nvme1n1.out‚33750.62109375 disk.sda.in‚3.546875 disk.sda.out‚741.62548828125 disk.rbd0.in‚0 disk.sdb1.out‚0 network.recv‚ 2.183972e+06 disk.dm-0.out‚98611.48828125 proc.cpu.threads‚17$ disk./.usageGB‚2710.647636413574 disk.nvme1n1.in‚ 743.53125 disk.rbd2.in‚0 disk.sdb.out‚741.62548828125 disk.nvme0n1.in‚ 939.5390625 disk.sda1.in‚0 disk.sdb1.in‚0$ memory_percent‚91.98713302612305" disk.nvme0n1.out‚ 64860.8671875 disk.rbd2.out‚ 1.908203125:ÿúì“é‚(:ˆ ÙÄéÔꬆm gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.157' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.78925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°Ô3}bê‚(:× ÙÄéÔ¨¢‘n disk.sdb2.out‚ 743.087890625 disk.sda1.in‚0$ memory_percent‚91.98906421661377* proc.memory.percent‚18.587851524353027 disk.sda.in‚3.546875 disk.sda.out‚ 743.087890625 disk.dm-0.in‚ 1683.0703125 cpu‚25.15372062258265# disk.nvme0n1.out‚64920.87109375 disk.sdb2.in‚ 0.6328125 disk.md0p1.in‚ 4.1796875 disk.dm-0.out‚98671.69921875 disk.md0.in‚ 4.1796875 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sdb.in‚ 0.6328125 disk.sdb.out‚ 743.087890625 disk.sda2.in‚3.546875 disk.sdb1.in‚0 network.sent‚ 1.433436e+07 disk.rbd0.out‚ 1.68359375 disk.md0p1.out‚ 714.82421875 proc.cpu.threads‚17$ disk./.usageGB‚2710.647636413574 disk.sda2.out‚ 743.087890625 disk.nvme1n1.in‚ 743.53125 disk.sdb1.out‚0 network.recv‚ 2.18858e+06) disk./.usagePercent‚75.79084240829583 disk.nvme0n1.in‚ 939.5390625! disk.nvme1n1.out‚ 33750.828125 disk.rbd0.in‚0) proc.memory.availableMB‚ 1312.51171875# proc.memory.rssMB‚ 3045.43359375 disk.md0.out‚ 714.82421875 disk.rbd2.out‚ 1.908203125²#³“ë‚(:ˆ èÄéÔùøþl gpu.0.gpu‚55.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚118.751' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.68775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù[96^ì‚(:Ó èÄéÔ²¦ãm$ disk./.usageGB‚2710.647636413574 disk.sdb.out‚ 744.93359375# disk.nvme1n1.out‚33750.99609375 disk.sdb2.in‚ 0.6328125 disk.sda1.out‚0# proc.memory.rssMB‚ 3045.43359375 disk.nvme1n1.in‚ 743.53125 disk.md0.in‚ 4.1796875 disk.rbd0.in‚0 disk.sdb1.out‚0$ memory_percent‚91.98949337005615) proc.memory.availableMB‚ 1312.44140625 disk.dm-0.out‚ 98886.3828125 disk.sdb.in‚ 0.6328125 disk.nvme0n1.in‚ 939.5390625* proc.memory.percent‚18.587851524353027 disk.md0p1.in‚ 4.1796875 disk.md0.out‚ 716.6171875 disk.sda.in‚3.546875 disk.sda.out‚ 744.93359375 disk.rbd0.out‚ 1.68359375 disk.sdb1.in‚0 network.sent‚ 1.434035e+07) disk./.usagePercent‚75.79084240829583 disk.dm-0.in‚ 1683.0703125 disk.sda2.in‚3.546875 proc.cpu.threads‚17 disk.md0p1.out‚ 716.6171875 disk.rbd2.in‚0 disk.sdb2.out‚ 744.93359375 disk.sda1.in‚0 cpu‚25.15341729370296 network.recv‚ 2.193239e+06 disk.sda2.out‚ 744.93359375# disk.nvme0n1.out‚65135.38671875 disk.rbd2.out‚ 1.908203125ÕbT“í‚(:ˆ ÷ÄéÔ—ñ÷l gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚173.953' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.48825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ!qChî‚(:Ý ÷ÄéÔèîím$ disk./.usageGB‚2710.117328643799 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.sdb.out‚746.3896484375 disk.sdb2.in‚ 0.6328125 disk.sda.out‚746.3896484375 disk.rbd0.in‚0) disk./.usagePercent‚75.77601478110188 disk.md0p1.in‚ 4.1796875 disk.sda2.in‚3.546875 disk.nvme0n1.in‚ 939.5390625# disk.nvme0n1.out‚65368.73046875 disk.md0.in‚ 4.1796875# proc.memory.rssMB‚ 3045.43359375 cpu‚25.153148974071748 disk.sdb1.in‚0 disk.md0p1.out‚ 718.0078125 disk.dm-0.in‚ 1683.0703125 disk.nvme1n1.in‚ 743.53125 disk.rbd2.in‚0* proc.memory.percent‚18.587851524353027 disk.sda2.out‚746.3896484375 disk.md0.out‚ 718.0078125 disk.sda.in‚3.546875 disk.sda1.in‚0 disk.rbd0.out‚ 1.68359375) proc.memory.availableMB‚ 1312.37109375# disk.nvme1n1.out‚33751.18359375 disk.sdb2.out‚746.3896484375 disk.dm-0.out‚ 99119.9140625 network.recv‚ 2.197296e+06 disk.sdb.in‚ 0.6328125 disk.rbd2.out‚ 1.908203125 disk.sda1.out‚0$ memory_percent‚91.98992252349854 network.sent‚ 1.4346186e+07†¨f2ï- _wandbruntime‚6714  _runtime‚6714áütÇðÁ _step‚468 & lr-AdamW/pg1‚0.0006614440988518892 & lr-AdamW/pg2‚0.0006614440988518892 trainer/global_step‚5749 % _timestamp‚1.7885026564554756e+09 _runtime‚6714.232812885²A&ÌñÆ & lr-AdamW/pg1‚0.0006614440988518892 & lr-AdamW/pg2‚0.0006614440988518892 trainer/global_step‚5749 % _timestamp‚1.7885026564554756e+09 _runtime‚6714.232812885 _step‚468Ôy.[€2ò- _wandbruntime‚6714  _runtime‚6714nÒÑAü'óöO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚539869192763904 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6573197362688 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6573197362688 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚489580528216064 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _runtime‚6714.233603588 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145408439 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚489580528216064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚631 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚633 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 _step‚469 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚467080398789552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚483007348856320 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚532998214569472 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚539869445486080 % _timestamp‚1.7885026564562762e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34799261 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚473629921563050 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚467080381750192 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6549522773498 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6548614798494 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚483007330853376 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71790422 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145382095 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6574107334144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 265239185 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6870978194432 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 trainer/global_step‚5749 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119830746 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119830746 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119830744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344Ù ‚àôûO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _runtime‚6714.233603588 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145408439 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚489580528216064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚631 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚633 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 _step‚469 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚467080398789552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚483007348856320 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚532998214569472 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚539869445486080 % _timestamp‚1.7885026564562762e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34799261 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚473629921563050 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚467080381750192 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6549522773498 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6548614798494 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚483007330853376 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71790422 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145382095 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6574107334144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 265239185 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6870978194432 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 trainer/global_step‚5749 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119830746 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119830746 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119830744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚539869192763904 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6573197362688 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6573197362688 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚489580528216064 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615ÕRäQˆ2õ- _wandbruntime‚6715  _runtime‚6715P)FÒ'ö€N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34805773 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265284202 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6575228807680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71802489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚483088967942144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚533088464214016 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145433296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚539960846360576 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106607652 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 145406926 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚489664196749824 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265284202 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119850906 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚467159416384096 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119850904 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6574318822912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 265257830 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚489664196749824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 trainer/global_step‚5749¸‹©æ '÷…N 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34805773 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265284202 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6575228807680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71802489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚483088967942144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚533088464214016 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145433296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚539960846360576 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106607652 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 145406926 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚489664196749824 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265284202 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119850906 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚467159416384096 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119850904 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6574318822912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 265257830 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚489664196749824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 trainer/global_step‚5749 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ÖÐö2ø- _wandbruntime‚6715  _runtime‚6715ÊÔ½Š\ùÖ 0loss/SRqqt1/detection_loss‚2.505584261919323 %loss/total_loss‚40.46473624538858 /loss/FRt2/detection_loss‚2.3078607210759507 /loss/FRt1/assignment_loss‚2.131452598222872 0loss/FRt3/assignment_loss‚1.4735872457104344 2loss/SRqqt2/assignment_loss‚1.6823548540472983 2loss/SRqqt4/assignment_loss‚1.6867571485658677 /loss/FRt4/detection_loss‚2.9746647820328223 1loss/SRqqt2/detection_loss‚2.1472781555993214 1loss/SRqqt3/detection_loss‚0.9610970392180928 /loss/FBt3/detection_loss‚2.1663855453953147 _runtime‚ 6715.24089995 0loss/FRt2/assignment_loss‚3.1362796107003854 trainer/global_step‚5749 0loss/FBt3/assignment_loss‚0.2149428897077712 1loss/FBt4/assignment_loss‚0.23709040119657793 .loss/FBt1/detection_loss‚0.776819414592215 /loss/FBt2/detection_loss‚2.1344178915023804 /loss/FRt4/assignment_loss‚3.345925231774648 1loss/SRqqt4/detection_loss‚2.4519280241085935 1loss/FBt1/assignment_loss‚0.11911258174147211 1loss/FBt2/assignment_loss‚0.15164745850488542 /loss/FRt3/detection_loss‚1.7067332587415172 /loss/FBt4/detection_loss‚2.0676043374197826 _step‚471 2loss/SRqqt1/assignment_loss‚1.3789518789241189 .loss/FRt1/detection_loss‚1.732849318198377 epoch‚4 % _timestamp‚1.7885026574638898e+09 2loss/SRqqt3/assignment_loss‚0.9734115964885608ØmØaúÛ _runtime‚ 6715.24089995 0loss/FRt2/assignment_loss‚3.1362796107003854 trainer/global_step‚5749 0loss/FBt3/assignment_loss‚0.2149428897077712 1loss/FBt4/assignment_loss‚0.23709040119657793 .loss/FBt1/detection_loss‚0.776819414592215 /loss/FBt2/detection_loss‚2.1344178915023804 /loss/FRt4/assignment_loss‚3.345925231774648 1loss/SRqqt4/detection_loss‚2.4519280241085935 1loss/FBt1/assignment_loss‚0.11911258174147211 1loss/FBt2/assignment_loss‚0.15164745850488542 /loss/FRt3/detection_loss‚1.7067332587415172 /loss/FBt4/detection_loss‚2.0676043374197826 _step‚471 2loss/SRqqt1/assignment_loss‚1.3789518789241189 .loss/FRt1/detection_loss‚1.732849318198377 epoch‚4 % _timestamp‚1.7885026574638898e+09 2loss/SRqqt3/assignment_loss‚0.9734115964885608 0loss/SRqqt1/detection_loss‚2.505584261919323 %loss/total_loss‚40.46473624538858 /loss/FRt2/detection_loss‚2.3078607210759507 /loss/FRt1/assignment_loss‚2.131452598222872 0loss/FRt3/assignment_loss‚1.4735872457104344 2loss/SRqqt2/assignment_loss‚1.6823548540472983 2loss/SRqqt4/assignment_loss‚1.6867571485658677 /loss/FRt4/detection_loss‚2.9746647820328223 1loss/SRqqt2/detection_loss‚2.1472781555993214 1loss/SRqqt3/detection_loss‚0.9610970392180928 /loss/FBt3/detection_loss‚2.1663855453953147× µ¥À“û‚(:ˆ †ÅéÔÏŽµl gpu.0.gpu‚55.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.219' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.55475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç­§Îlü‚(:á †ÅéÔÄÑÍl disk.sda.in‚3.546875 disk.sdb.in‚ 0.6328125 disk.md0.out‚ 719.38671875 disk.rbd0.out‚ 1.68359375 disk.sdb1.in‚0 disk.sda.out‚747.84716796875 disk.sda1.in‚0# proc.memory.rssMB‚ 3045.43359375 network.sent‚ 1.4350428e+07# disk.nvme0n1.out‚65499.44140625 disk.nvme1n1.in‚ 743.53125 disk.rbd0.in‚0 disk.md0p1.out‚ 719.38671875! disk.sda2.out‚747.84716796875$ disk./.usageGB‚2710.067226409912 disk.nvme0n1.in‚ 939.5390625 disk.md0p1.in‚ 4.1796875 disk.dm-0.out‚ 99251.515625* proc.memory.percent‚18.587851524353027 cpu‚25.152856089194138 disk.sdb2.in‚ 0.6328125 disk.rbd2.out‚ 1.908203125 disk.sda1.out‚0 disk.dm-0.in‚ 1683.0703125 disk.sdb.out‚747.84716796875 disk.sda2.in‚3.546875$ memory_percent‚91.99328422546387# disk.nvme1n1.out‚33752.07421875 disk.rbd2.in‚0! disk.sdb2.out‚747.84716796875 disk.sdb1.out‚0( proc.memory.availableMB‚ 1311.8203125 proc.cpu.threads‚17 network.recv‚ 2.202212e+06) disk./.usagePercent‚75.77461390167298 disk.md0.in‚ 4.1796875Ï4ÕÂ’ý‚(:‡ •ÅéÔáØ«m gpu.0.gpu‚16.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚179.674' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.9185 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{¸Š hþ‚(:Ý •ÅéÔí¯·m' proc.memory.availableMB‚ 1311.828125 cpu‚25.152546657322848 disk.sda1.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 721.44140625 disk.sda.in‚3.546875 disk.dm-0.out‚ 99489.484375) disk./.usagePercent‚75.77679532287593$ memory_percent‚91.99323654174805 disk.sdb2.in‚ 0.6328125 disk.sda1.out‚0 disk.rbd0.out‚ 1.68359375 disk.sdb.out‚749.9814453125 disk.md0.in‚ 4.1796875 disk.rbd2.in‚0 disk.sda.out‚749.9814453125 disk.md0p1.in‚ 4.1796875 disk.md0p1.out‚ 721.44140625* proc.memory.percent‚18.587851524353027 proc.cpu.threads‚17 network.sent‚ 1.4457603e+07 network.recv‚ 2.206514e+06# disk.nvme0n1.out‚65656.68359375# disk.nvme1n1.out‚33832.80078125 disk.sdb2.out‚749.9814453125 disk.sdb1.in‚0 disk.dm-0.in‚ 1683.0703125# proc.memory.rssMB‚ 3045.43359375% disk./.usageGB‚2710.1452445983887 disk.sda2.in‚3.546875 disk.sda2.out‚749.9814453125 disk.nvme0n1.in‚ 939.5390625 disk.sdb.in‚ 0.6328125 disk.nvme1n1.in‚ 743.53125 disk.rbd2.out‚ 1.908203125 disk.rbd0.in‚0‚+ZÜ‘ÿ‚(:† ¤ÅéÔõÜêk gpu.0.gpu‚18.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.916' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.229 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€aÈ€‚(:Ù ¤ÅéÔ×ͬm disk.nvme1n1.in‚ 743.53125 disk.sdb2.in‚ 0.6328125 disk.sda.out‚ 751.470703125 disk.sdb1.out‚0 disk.sda2.out‚ 751.470703125 network.sent‚ 1.4462792e+07 disk.sda.in‚3.546875 disk.dm-0.out‚99640.23828125# disk.nvme0n1.out‚65804.79296875) disk./.usagePercent‚75.77686283899277 disk.rbd0.out‚ 1.68359375 disk.sda2.in‚3.546875 disk.md0.out‚ 722.87109375 disk.rbd2.out‚ 1.908203125 cpu‚25.15228347733219$ memory_percent‚91.99166297912598( proc.memory.availableMB‚ 1312.0859375" disk.nvme1n1.out‚ 33835.4453125 disk.md0.in‚ 4.1796875$ disk./.usageGB‚2710.147659301758 disk.sda1.in‚0 disk.sdb1.in‚0 @Dqœdisk.dm-0.in‚ 1683.0703125 disk.sdb.out‚ 751.470703125 disk.nvme0n1.in‚ 939.5390625 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd2.in‚0# proc.memory.rssMB‚ 3045.43359375 disk.md0p1.in‚ 4.1796875 disk.sdb.in‚ 0.6328125 network.recv‚ 2.210921e+06 disk.sdb2.out‚ 751.470703125 disk.md0p1.out‚ 722.87109375* proc.memory.percent‚18.587851524353027 proc.cpu.threads‚17áÍû²“‚(:ˆ ³ÅéÔðßþl gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚125.603' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.40075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØÜŽÈl‚‚(:á ³ÅéÔŒ¥ùm% disk./.usageGB‚2710.2739295959473! disk.sda2.out‚752.94970703125 disk.md0.in‚ 4.1796875! disk.sdb2.out‚752.94970703125 disk.rbd0.in‚0) proc.memory.availableMB‚ 1312.30859375 disk.md0p1.in‚ 4.1796875 disk.sdb.in‚ 0.6328125 disk.sda2.in‚3.546875 disk.nvme0n1.in‚ 939.5390625 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.md0p1.out‚ 724.29296875 disk.sda1.out‚0# proc.memory.rssMB‚ 3045.43359375* proc.memory.percent‚18.587851524353027 network.recv‚ 2.215033e+06 disk.sdb.out‚752.94970703125 disk.sda.in‚3.546875 disk.rbd0.out‚ 1.68359375 disk.sda.out‚752.94970703125 network.sent‚ 1.44687e+07 disk.dm-0.in‚ 1683.0703125# disk.nvme0n1.out‚65918.85546875# disk.nvme1n1.out‚33894.29296875 disk.sdb2.in‚ 0.6328125 disk.sda1.in‚0 disk.nvme1n1.in‚ 743.53125 disk.md0.out‚ 724.29296875 disk.rbd2.out‚ 1.908203125) disk./.usagePercent‚75.78039340926654 disk.sdb1.out‚0 disk.dm-0.out‚ 99813.1484375# memory_percent‚91.9903039932251 cpu‚25.152015999601094 disk.rbd2.in‚0Ðÿí>2ƒ- _wandbruntime‚6768  _runtime‚6768SM³Å„¿ _step‚472 % lr-AdamW/pg1‚0.000654089586921424 % lr-AdamW/pg2‚0.000654089586921424 trainer/global_step‚5799 % _timestamp‚1.7885027107743917e+09 _runtime‚6768.551665267<œMÊ…Ä _step‚472 % lr-AdamW/pg1‚0.000654089586921424 % lr-AdamW/pg2‚0.000654089586921424 trainer/global_step‚5799 % _timestamp‚1.7885027107743917e+09 _runtime‚6768.551665267ØLfFá2†- _wandbruntime‚6768  _runtime‚6768€¬=#ü'‡öO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚493717569932288 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚493718497906688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35091787 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚6768.552646522 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6605400028606 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72393672 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚537509499791872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚471031278516752 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146626997 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚544439458884096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6630194759168 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107485461 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6630194759168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146653341 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚493717569932288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚487088285144576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚645 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚493718497906688 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6629284787712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚487088303147520 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120838744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚487088303147520 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6929935152128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚477636678545358 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120838744 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚471031261477392mÒ^(ˆûO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146626997 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚544439458884096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6630194759168 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107485461 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6630194759168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146653341 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚493717569932288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚487088285144576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚645 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚493718497906688 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6629284787712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚487088303147520 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120838744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚487088303147520 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6929935152128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚477636678545358 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120838744 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚471031261477392 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚493717569932288 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚493718497906688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35091787 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚6768.552646522 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6605400028606 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72393672 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚537509499791872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚471031278516752ÙyÁ«H2‰- _wandbruntime‚6769  _runtime‚6769.¿5 ö%Š€N F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107502524 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120858904 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146678198 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6606517302092 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚471110296111296 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72405739 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 267510732 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107503175 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6631316232704 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267537104 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120858906 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146678198 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚537599725496320 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6631316232704 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120858906 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚651 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚544530841288704 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚487169904230400 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146651828 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6605609326880 _step‚474 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚493800310478336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚537599749436416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚487169904230400 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35096787 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚493801238466048 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120858904 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚5799 _runtime‚6769.536886476 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6630406247936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35097436 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚477715888398816µ‹€V ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚471110279071936 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144½W?, '‹…N JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 267510732 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107503175 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6631316232704 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267537104 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120858906 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146678198 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚537599725496320 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6631316232704 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120858906 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚651 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚544530841288704 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚487169904230400 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146651828 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6605609326880 _step‚474 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚493800310478336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚537599749436416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚487169904230400 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35096787 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚493801238466048 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120858904 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚5799 _runtime‚6769.536886476 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6630406247936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35097436 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚477715888398816 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚471110279071936 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107502524 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120858904 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146678198 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6606517302092 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚471110296111296 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72405739 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570ÚåX—2Œ- _wandbruntime‚6769  _runtime‚6769Ñ÷ÉpYÓ 1loss/SRqqt3/detection_loss‚0.9744073728677528 0loss/SRqqt4/detection_loss‚2.544364875760572 .loss/FBt2/detection_loss‚2.134024115403493 .loss/FBt3/detection_loss‚2.230929434299469 epoch‚4 2loss/SRqqt4/assignment_loss‚1.6038690574210266 1loss/FBt1/assignment_loss‚0.20542500627418112 /loss/FRt4/detection_loss‚3.0270657782201416 /loss/FBt1/detection_loss‚0.6673458092124529 0loss/FBt2/assignment_loss‚0.2623622350673637 1loss/SRqqt1/detection_loss‚2.2089855428095215 0loss/SRqqt2/detection_loss‚2.672613764631337 /loss/FRt1/detection_loss‚1.9188169712424976 /loss/FRt3/detection_loss‚1.7960312062641606 0loss/FRt1/assignment_loss‚1.8713299161509465 /loss/FRt4/assignment_loss‚3.548726479212443 2loss/SRqqt2/assignment_loss‚1.8319941211877198 /loss/FBt4/detection_loss‚3.5306312143802643 _runtime‚6769.538472504 /loss/FRt2/assignment_loss‚3.478986021949024 1loss/SRqqt1/assignment_loss‚1.737115153559932 1loss/FBt3/assignment_loss‚0.11144844408199361 _step‚475 2loss/SRqqt3/assignment_loss‚1.0831305612689526 %loss/total_loss‚44.11622995377019 trainer/global_step‚5799 .loss/FRt2/detection_loss‚2.254893224413802 0loss/FRt3/assignment_loss‚1.7348415469750762 0loss/FBt4/assignment_loss‚0.6868921011160637 % _timestamp‚1.7885027117614157e+09DÚKg^ŽØ 0loss/FRt1/assignment_loss‚1.8713299161509465 /loss/FRt4/assignment_loss‚3.548726479212443 2loss/SRqqt2/assignment_loss‚1.8319941211877198 /loss/FBt4/detection_loss‚3.5306312143802643 _runtime‚6769.538472504 /loss/FRt2/assignment_loss‚3.478986021949024 1loss/SRqqt1/assignment_loss‚1.737115153559932 1loss/FBt3/assignment_loss‚0.11144844408199361 _step‚475 2loss/SRqqt3/assignment_loss‚1.0831305612689526 %loss/total_loss‚44.11622995377019 trainer/global_step‚5799 .loss/FRt2/detection_loss‚2.254893224413802 0loss/FRt3/assignment_loss‚1.7348415469750762 0loss/FBt4/assignment_loss‚0.6868921011160637 % _timestamp‚1.7885027117614157e+09 1loss/SRqqt3/detection_loss‚0.9744073728677528 0loss/SRqqt4/detection_loss‚2.544364875760572 .loss/FBt2/detection_loss‚2.134024115403493 .loss/FBt3/detection_loss‚2.230929434299469 epoch‚4 2loss/SRqqt4/assignment_loss‚1.6038690574210266 1loss/FBt1/assignment_loss‚0.20542500627418112 /loss/FRt4/detection_loss‚3.0270657782201416 /loss/FBt1/detection_loss‚0.6673458092124529 0loss/FBt2/assignment_loss‚0.2623622350673637 1loss/SRqqt1/detection_loss‚2.2089855428095215 0loss/SRqqt2/detection_loss‚2.672613764631337 /loss/FRt1/detection_loss‚1.9188169712424976 /loss/FRt3/detection_loss‚1.7960312062641606Ûzy’‚(:‡ ÂÅéÔ÷ù¡m gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚174.726' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.6815 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XÓ8Ì]‚(:Ò ÂÅéÔè¶õn network.recv‚ 2.219638e+06 disk.sda2.out‚ 754.7578125 disk.rbd2.in‚0 disk.sda.out‚ 754.7578125 disk.md0p1.in‚ 4.1796875# memory_percent‚91.9905424118042! disk.nvme0n1.out‚ 65919.515625 disk.nvme1n1.in‚ 743.53125 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sdb.out‚ 754.7578125# disk.nvme1n1.out‚33953.58203125 disk.sdb1.in‚0 disk.md0p1.out‚ 726.05078125) proc.memory.availableMB‚ 1312.26953125% disk./.usageGB‚2710.3991775512695 disk.nvme0n1.in‚ 939.5390625 disk.dm-0.in‚ 1683.0703125) disk./.usagePercent‚75.78389539451769 disk.sdb.in‚ 0.6328125 disk.sda2.in‚3.546875 disk.sdb2.out‚ 754.7578125 proc.cpu.threads‚17 disk.md0.in‚ 4.1796875 disk.sdb2.in‚ 0.6328125 cpu‚25.151746861647986 network.sent‚ 1.4487877e+07 disk.rbd2.out‚ 1.908203125 disk.rbd0.out‚ 1.68359375 disk.sdb1.out‚0 disk.dm-0.out‚99873.09765625* proc.memory.percent‚18.587851524353027 disk.md0.out‚ 726.05078125 disk.sda.in‚3.546875 disk.sda1.in‚0# proc.memory.rssMB‚ 3045.43359375xÔ­’‘‚(:‡ ÑÅéÔÏëÔk gpu.0.gpu‚7.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚185.149' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚46.28725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B^"d’‚(:Ù ÑÅéÔ“Áƒm) disk./.usagePercent‚75.79087109997897% disk./.usageGB‚2710.6486625671387 disk.sdb1.out‚0 disk.dm-0.in‚ 1683.0703125 disk.sda2.out‚756.4931640625 disk.nvme0n1.in‚ 939.5390625 disk.md0.in‚ 4.1796875 network.sent‚ 1.4581819e+07 network.recv‚ 2.223936e+06 disk.sda2.in‚3.546875 disk.rbd2.out‚ 1.908203125 proc.cpu.threads‚17 disk.rbd0.out‚ 1.68359375 disk.nvme1n1.in‚ 743.53125* proc.memory.percent‚18.587851524353027 disk.sda.out‚756.4931640625! disk.nvme0n1.out‚ 66233.984375 disk.sdb2.in‚ 0.6328125 disk.sda.in‚3.546875 disk.sda1.in‚0 disk.sdb1.in‚0 disk.md0p1.in‚ 4.1796875 disk.rbd2.in‚0) proc.memory.availableMB‚ 1312.00390625 disk.sdb2.out‚756.4931640625 disk.dm-0.out‚ 100334.296875 disk.sdb.in‚ 0.6328125 disk.sdb.out‚756.4931640625 disk.nvme1n1.out‚ 34100.3125$ memory_percent‚91.99216365814209 cpu‚25.15150105798777 disk.sda1.out‚0 disk.rbd0.in‚0 disk.md0p1.out‚ 727.71484375 disk.md0.out‚ 727.71484375# proc.memory.rssMB‚ 3045.43359375k,œ“‚(:‘ àÅéÔ¦Ñèk gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚117.81' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.452499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zÂ`k”‚(:à àÅéÔ¢‚m# proc.memory.rssMB‚ 3045.43359375% disk./.usageGB‚2710.6486625671387 disk.sdb2.in‚ 0.6328125 disk.sda.in‚3.546875! disk.dm-0.out‚100526.57421875 disk.sda2.in‚3.546875 network.recv‚ 2.228213e+06 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚757.8017578125" disk.nvme0n1.out‚ 66382.0234375$ memory_percent‚91.99204444885254( proc.memory.availableMB‚ 1312.0234375 disk.rbd0.out‚ 1.68359375 disk.rbd2.out‚ 1.908203125 cpu‚25.151241806112722 network.sent‚ 1.4587851e+07 disk.md0p1.in‚ 4.1796875 disk.nvme0n1.in‚ 939.5390625 disk.md0.in‚ 4.1796875 disk.sdb2.out‚757.8017578125 disk.sda1.out‚0 disk.sdb1.out‚0 disk.sdb.out‚757.8017578125 disk.nvme1n1.in‚ 743.53125 disk.md0.out‚ 728.95703125* proc.memory.percent‚18.587851524353027 disk.sda.out‚757.8017578125 disk.sdb1.in‚0 disk.md0p1.out‚ 728.95703125 disk.sdb.in‚ 0.6328125 disk.rbd2.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚75.79087109997897 disk.dm-0.in‚ 1683.0703125# disk.nvme1n1.out‚34144.55078125§ó%2•- _wandbruntime‚6822  _runtime‚6822]&]Ç–Á % _timestamp‚1.7885027651489575e+09 _runtime‚6822.926284987 _step‚476 & lr-AdamW/pg1‚0.0006467084181719907 & lr-AdamW/pg2‚0.0006467084181719907 trainer/global_step‚5849ÈèÑÌ—Æ & lr-AdamW/pg2‚0.0006467084181719907 trainer/global_step‚5849 % _timestamp‚1.7885027651489575e+09 _runtime‚6822.926284987 _step‚476 & lr-AdamW/pg1‚0.0006467084181719907Ü%î¿[2˜- _wandbruntime‚6822  _runtime‚6822ü•¶Ûü'™öO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚491169239435776 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108380337 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 147898243 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚549009302194176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6988493239808 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35383413 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6661277283714 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚491169239435776 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121846744 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚542020808954368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6685372212736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚497854611648512 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚474982158243952 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6660369308710 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497854611648512 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6686282184192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚542020785014272 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚549009051569152 _step‚477 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 147871899 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚474982141204592 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 108379702 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _runtime‚6822.927572003 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35382780 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6686282184192 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6988266554880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 269718643 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121846744 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6685372212736 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 trainer/global_step‚5849 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚635 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72996924´wǦšûO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚491169257438720 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚491169257438720 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 147898243 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚549009302194176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6988493239808 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35383413 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6661277283714 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚491169239435776 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121846744 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚542020808954368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6685372212736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚497854611648512 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚474982158243952 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6660369308710 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497854611648512 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6686282184192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚542020785014272 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚549009051569152 _step‚477 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 147871899 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚474982141204592 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 108379702 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _runtime‚6822.927572003 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35382780 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6686282184192 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6988266554880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 269718643 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121846744 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6685372212736 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 trainer/global_step‚5849 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚635 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72996924 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚491169239435776 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108380337 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147923100 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147896730 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 147896730 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35389227 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚542111010718720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚475061175838496 % _timestamp‚1.7885027662001617e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚613 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚481723570395696 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚491250876524544 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6686493672960 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147923100 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108397601 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6662394557200 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚478 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73008987 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269790006 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121866904 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73008989 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 269763634 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147923100 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147896730 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 147896730 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35389227 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚542111010718720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚475061175838496 % _timestamp‚1.7885027662001617e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚613 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚481723570395696 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚491250876524544 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6686493672960 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147923100 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108397601 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6662394557200 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚478 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73008987 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269790006 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121866904 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73008989 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 269763634 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚546532094176768 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149143145 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚501992581339136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122854746 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚6877.526026681 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚501991653364736 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚501991653364736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35674831 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35675456 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 271997891 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚485649267495610 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚546532070236672 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚478933037971152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚478933020931792 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚501992581339136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚485650192509974 % _timestamp‚1.7885028197486575e+09 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122854746 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7047220145152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6716246563818 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 trainer/global_step‚5899 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚495250193726976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6741459637760 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚553579057405440 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6742369609216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73600174 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122854744 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚495250211729920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149143145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109275630 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6717154538822 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 È(®ûO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35675456 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 271997891 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚485649267495610 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚546532070236672 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚478933037971152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚478933020931792 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚501992581339136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚485650192509974 % _timestamp‚1.7885028197486575e+09 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122854746 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7047220145152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6716246563818 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 trainer/global_step‚5899 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚495250193726976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6741459637760 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚553579057405440 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6742369609216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73600174 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122854744 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚495250211729920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149143145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109275630 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6717154538822 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚546532094176768 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149143145 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚501992581339136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122854746 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚6877.526026681 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚501991653364736 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚501991653364736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35674831áXwðt2¯- _wandbruntime‚6878  _runtime‚6878³.6ßÁ$°€N B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚495331812812800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214088704 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚485730327378004 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 % _timestamp‚1.7885028207849705e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149168002 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122874904 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚485729402363432 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 149168002 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚553670475943936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚614 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122874906 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149141632 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5899 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 272016536 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 272042908 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109292885 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚495331830815744 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6743491082752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6743491082752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 272016536 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122874906 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35680648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6717363837096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _runtime‚6878.563486057 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6742581097984 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚479012038526336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122874904 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7048394091520 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚479012055565696 _step‚482 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_byteýúüEs.all.current‚ 238028800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149141632 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚553670713972736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚502074393910784 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5899 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 272016536 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 272042908 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109292885 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚495331830815744 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6743491082752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6743491082752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 272016536 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122874906 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35680648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6717363837096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _runtime‚6878.563486057 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6742581097984 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚479012038526336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122874904 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7048394091520 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚479012055565696 _step‚482 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238028800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149141632 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚553670713972736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚502074393910784 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚485729402363432 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 149168002 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚553670475943936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚614 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122874906 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149141632 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7105963239936 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚482883917698352 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚489656024477918 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 274224447 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35967751 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚551043379399168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7885028743819153e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 274250793 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6797547062784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚499331166021120 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150361703 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 110171175 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 _step‚485 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6798457034240 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123862746 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74203422 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6772123818926 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35967102 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _runtime‚6932.159199809 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚482883900658992 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚506128695080960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚651 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6773031793930 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74203424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚489656949492282 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚506128695080960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚558149094111232 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚558149342639104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 110170524 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7105738652160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6798457034240 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6797547062784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 trainer/global_step‚5949 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123862744 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚506129623055360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150361703 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚499331148018176 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚649 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚499331166021120 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150361703 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 110171175 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 _step‚485 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6798457034240 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123862746 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74203422 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_p¿™™ool.current‚ 224587776 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6772123818926 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35967102 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _runtime‚6932.159199809 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚482883900658992 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚506128695080960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚651 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6773031793930 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74203424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚489656949492282 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚506128695080960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚558149094111232 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚558149342639104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 110170524 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7105738652160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6798457034240 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6797547062784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 trainer/global_step‚5949 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123862744 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚506129623055360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150361703 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚499331148018176 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚649 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚499331166021120 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7105963239936 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚482883917698352 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚489656024477918 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 274224447 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35967751 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚551043379399168åýäY¥2Ã- _wandbruntime‚6933  _runtime‚6933ù‰òš'Ä€N \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6799578507776 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6798668523008 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚482962918253536 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _step‚486 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123882904 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110188332 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216185856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150412904 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6774149067416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚6933.260932765 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚499412785106944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚489737084360312 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6799578507776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 274295810 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123882904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚551133605103616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚499412767104000 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6773241092204 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123882906 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚506212363614720 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150386534 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 274295810 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7106930225664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 110188960 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚489736159345740 trainer/global_step‚5949 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150386534 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7107146411520 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521£êv 'Å…N ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 274295810 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123882904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚551133605103616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚499412767104000 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6773241092204 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123882906 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚506212363614720 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150386534 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 274295810 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7106930225664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 110188960 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚489736159345740 trainer/global_step‚5949 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150386534 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7107146411520 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6799578507776 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6798668523008 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚482962918253536 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _step‚486 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123882904 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110188332 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216185856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150412904 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6774149067416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚6933.260932765 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚499412785106944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚489737084360312 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6799578507776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628æa}òz2Æ- _wandbruntime‚6933  _runtime‚6933iLù‚]Ç× 0loss/SRqqt4/detection_loss‚2.629878563915982 /loss/FBt2/detection_loss‚2.1963837862014772 .loss/FRt1/detection_loss‚1.675112584738859 1loss/SRqqt2/detection_loss‚2.5575731019179027 /loss/FBt4/detection_loss‚2.1582882925868034 trainer/global_step‚5949 0loss/FRt2/assignment_loss‚2.9860264129108853 2loss/SRqqt2/assignment_loss‚1.6742479478319485 /loss/FRt4/assignment_loss‚2.775491926981055 2loss/SRqqt1/assignment_loss‚1.2257284643989503 /loss/FRt3/detection_loss‚1.9313377470243722 1loss/FBt2/assignment_loss‚0.16519560795277358 0loss/FBt3/assignment_loss‚0.2020080691436116 /loss/FRt4/detection_loss‚3.1868675636208574 1loss/SRqqt3/detection_loss‚1.1861520147421092 /loss/FBt1/detection_loss‚0.7280432772641252 _runtime‚6933.263752678 _step‚487 2loss/SRqqt3/assignment_loss‚1.0871571917347957 2loss/SRqqt4/assignment_loss‚1.8501324200016611 1loss/FBt1/assignment_loss‚0.11288438246364947 epoch‚4 0loss/FRt1/assignment_loss‚2.1916050667665443 1loss/SRqqt1/detection_loss‚2.0857874117791653 /loss/FBt3/detection_loss‚2.9506969352563224 % _timestamp‚1.7885028754864445e+09 0loss/FRt3/assignment_loss‚2.0306758116930723 0loss/FBt4/assignment_loss‚0.3578746995164579 .loss/FRt2/detection_loss‚2.373867705464363 %loss/total_loss‚42.31901698590775á S bÈÜ /loss/FRt4/assignment_loss‚2.775491926981055 2loss/SRqqt1/assignment_loss‚1.2257284643989503 /loss/FRt3/detection_loss‚1.9313377470243722 1loss/FBt2/assignment_loss‚0.16519560795277358 0loss/FBt3/assignment_loss‚0.2020080691436116 /loss/FRt4/detection_loss‚3.1868675636208574 1loss/SRqqt3/detection_loss‚1.1861520147421092 /loss/FBt1/detection_loss‚0.7280432772641252 _runtime‚6933.263752678 _step‚487 2loss/SRqqt3/assignment_loss‚1.0871571917347957 2loss/SRqqt4/assignment_loss‚1.8501324200016611 1loss/FBt1/assignment_loss‚0.11288438246364947 epoch‚4 0loss/FRt1/assignment_loss‚2.1916050667665443 1loss/SRqqt1/detection_loss‚2.0857874117791653 /loss/FBt3/detection_loss‚2.9506969352563224 % _timestamp‚1.7885028754864445e+09 0loss/FRt3/assignment_loss‚2.0306758116930723 0loss/FBt4/assignment_loss‚0.3578746995164579 .loss/FRt2/detection_loss‚2.373867705464363 %loss/total_loss‚42.31901698590775 0loss/SRqqt4/detection_loss‚2.629878563915982 /loss/FBt2/detection_loss‚2.1963837862014772 .loss/FRt1/detection_loss‚1.675112584738859 1loss/SRqqt2/detection_loss‚2.5575731019179027 /loss/FBt4/detection_loss‚2.1582882925868034 trainer/global_step‚5949 0loss/FRt2/assignment_loss‚2.9860264129108853 2loss/SRqqt2/assignment_loss‚1.6742479478319485çG­Ø,“É‚(:ˆ çÆéÔìîÞm gpu.0.gpu‚14.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.501' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.37525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ܯZYkÊ‚(:à çÆéÔ¸î™n$ memory_percent‚91.98400974273682# proc.memory.rssMB‚ 3045.43359375* proc.memory.percent‚18.587851524353027 disk.sda2.out‚772.9599609375 disk.md0.in‚ 4.1796875 network.sent‚ 1.4844453e+07% disk./.usageGB‚2710.3971633911133 disk.sdb2.in‚ 0.6328125 disk.sda.in‚3.546875 disk.rbd0.out‚ 1.68359375 disk.md0.out‚ 743.55859375) disk./.usagePercent‚75.78383907775672 disk.sdb.out‚772.9599609375 disk.rbd2.in‚0 network.recv‚ 2.268518e+06 disk.rbd0.in‚0! disk.dm-0.out‚102435.94140625! disk.nvme1n1.out‚ 34475.015625 disk.sdb2.out‚772.9599609375 disk.md0p1.in‚ 4.1796875 disk.md0p1.out‚ 743.55859375 disk.sdb.in‚ 0.6328125# disk.nvme0n1.out‚67960.92578125 disk.sda1.out‚0 disk.nvme1n1.in‚ 743.53125 proc.cpu.threads‚17 disk.sda.out‚772.9599609375 disk.sda1.in‚0 disk.sdb1.out‚0 disk.sda2.in‚3.546875 disk.nvme0n1.in‚ 939.5390625) proc.memory.availableMB‚ 1313.33984375 cpu‚25.148805751207238 disk.sdb1.in‚0 disk.dm-0.in‚ 1683.0703125 disk.rbd2.out‚ 1.908203125[¹ï’Ë‚(:‡ öÆéÔ—¶Ám gpu.0.gpu‚7.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.059' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.26475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô/VVÌ‚(:Ø öÆéÔ÷…ún disk.sdb.out‚774.3115234375$ memory_percent‚91.98548î2A7 793792725 disk.sda.in‚3.546875 disk.md0p1.out‚ 744.8515625 disk.sda2.out‚774.3115234375# proc.memory.rssMB‚ 3045.43359375 disk.nvme1n1.in‚ 743.53125 disk.md0.in‚ 4.1796875 disk.rbd2.out‚ 1.908203125 disk.sda.out‚774.3115234375 cpu‚25.148480591555966 disk.sda1.out‚0 disk.rbd0.in‚0 disk.md0p1.in‚ 4.1796875 network.sent‚ 1.495453e+07% disk./.usageGB‚2710.6471633911133! disk.nvme0n1.out‚ 68220.984375 disk.sdb2.in‚ 0.6328125 disk.rbd0.out‚ 1.68359375) proc.memory.availableMB‚ 1313.09765625 disk.nvme0n1.in‚ 939.5390625! disk.nvme1n1.out‚ 34475.234375 disk.md0.out‚ 744.8515625 disk.dm-0.in‚ 1683.0703125* proc.memory.percent‚18.587851524353027 disk.sdb.in‚ 0.6328125 disk.sdb2.out‚774.3115234375) disk./.usagePercent‚75.79082918238986 disk.sdb1.in‚0 disk.dm-0.out‚ 102696.21875 disk.sda2.in‚3.546875 proc.cpu.threads‚17 network.recv‚ 2.274269e+06 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sdb1.out‚0Mb›G’Í‚(:‡ …ÇéÔ×îÜk gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó7Æj΂(:ß …ÇéÔ‹õªm disk.md0.out‚ 746.26171875 disk.rbd2.in‚0 disk.sdb.out‚775.78076171875 cpu‚25.148103150383882 disk.rbd2.out‚ 1.908203125! disk.sda2.out‚775.78076171875* proc.memory.percent‚18.587851524353027% disk./.usageGB‚2710.6471633911133 disk.rbd0.in‚0! disk.dm-0.out‚102891.30078125 disk.sdb1.in‚0 disk.dm-0.in‚ 1683.0703125 network.recv‚ 2.277814e+06 disk.md0.in‚ 4.1796875 disk.sda.in‚3.546875 disk.rbd0.out‚ 1.68359375 disk.sdb.in‚ 0.6328125 disk.sda2.in‚3.546875! disk.sdb2.out‚775.78076171875 network.sent‚ 1.4959777e+07 disk.nvme0n1.in‚ 939.5390625 disk.sda.out‚775.78076171875# proc.memory.rssMB‚ 3045.43359375 proc.cpu.threads‚17) disk./.usagePercent‚75.79082918238986 disk.sdb2.in‚ 0.6328125 disk.sda1.out‚0 disk.md0p1.in‚ 4.1796875 disk.md0p1.out‚ 746.26171875$ proc.memory.availableMB‚1312.375 disk.sdb1.out‚0! disk.nvme0n1.out‚ 68415.859375 disk.nvme1n1.in‚ 743.53125# disk.nvme1n1.out‚34475.44140625 disk.sda1.in‚0$ memory_percent‚91.98989868164062$•^2Ï- _wandbruntime‚6989  _runtime‚6989|O§™ÇÐÁ trainer/global_step‚5999 % _timestamp‚1.7885029321976771e+09 _runtime‚6989.975255545 _step‚488 & lr-AdamW/pg1‚0.0006244203791066577 & lr-AdamW/pg2‚0.0006244203791066577¥d®ÌÑÆ % _timestamp‚1.7885029321976771e+09 _runtime‚6989.975255545 _step‚488 & lr-AdamW/pg1‚0.0006244203791066577 & lr-AdamW/pg2‚0.0006244203791066577 trainer/global_step‚5999èí f¸2Ò- _wandbruntime‚6989  _runtime‚6989ÄjÆñü'ÓöO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124870746 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚503412120312320 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124870744 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚645 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚510265736797184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6854544459264 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 111066275 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151632949 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚489 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36259601 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚5999 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚643 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 276477349 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 151606605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74806672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚510266664771584 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124870746 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 111065630 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚562719254380544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 % _timestamp‚1.7885029321991038e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6853634487808 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚493663706474590 _runtime‚6989.976811853 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚503412120312320 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151606605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36258958 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124870744 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7164613699072 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6828001074034 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6853634487808 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚555554664621568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6828909049038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74806674 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚555554640681472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚562719500811264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚503412102309376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚503412102309376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 276503695 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151632949 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚493662781460226 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚486834780386192 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚510265736797184 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6854544459264 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144÷m¢:(ÔûO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 276477349 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 151606605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74806672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚510266664771584 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124870746 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 111065630 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚562719254380544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 % _timestamp‚1.7885029321991038e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6853634487808 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚493663706474590 _runtime‚6989.976811853 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚503412120312320 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151606605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36258958 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124870744 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7164613699072 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6828001074034 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6853634487808 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚555554664621568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6828909049038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74806674 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚555554640681472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚562719500811264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚503412102309376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚503412102309376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 276503695 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151632949 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚493662781460226 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚486834780386192 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚510265736797184 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6854544459264 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124870746 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚503412120312320 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124870744 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚645 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚510265736797184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6854544459264 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 111066275 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151632949 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚489 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36259601 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚5999 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚643 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192é…°WV“Õ‚(:ˆ ”ÇéÔ‘ß½n gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ·®wTÖ‚(:É ”ÇéÔµ†Ón# memory_percent‚91.9898271560669* proc.memory.percent‚18.587851524353027 disk.sdb2.out‚ 777.888671875 disk.sda.in‚3.546875 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sdb2.in‚ 0.6328125 disk.nvme0n1.in‚ 939.5390625 disk.sdb.in‚ 0.6328125 disk.sdb.out‚ 777.888671875 disk.sda2.out‚ 777.888671875 disk.nvme1n1.in‚ 743.53125 disk.nvme1n1.out‚ 34475.65625) proc.memory.availableMB‚ 1312.38671875 disk.rbd0.in‚0 disk.md0.in‚ 4.1796875 disk.rbd2.in‚0% disk./.usageGB‚2710.6471633911133 disk.md0p1.out‚748.3125 cpu‚25.147812239925788 network.sent‚ 1.4963618e+07 disk.sda1.out‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚ 4.1796875 disk.dm-0.out‚ 103247.75 disk.sda.out‚ 777.888671875 disk.md0.out‚748.3125 disk.rbd2.out‚ 1.908203125# proc.memory.rssMB‚ 3045.43359375 network.recv‚ 2.282386e+06) disk./.usagePercent‚75.79082918238986 disk.rbd0.out‚ 1.68359375 disk.dm-0.in‚ 1683.0703125 disk.sda2.in‚3.546875 disk.nvme0n1.out‚ 68772.09375.y r2×- _wandbruntime‚6991  _runtime‚6991tªv>ØÿM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6829118347312 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124890904 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6855665932800 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151657806 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7166009273856 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚486913797980736 trainer/global_step‚5999 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 111084038 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124890906 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚555644866385920 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚493742916328048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚503493721395200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276548712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚503493721395200 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 276522340 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 $ _timestamp‚1.788502933366975e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36265299 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124890906 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151631436 _runtime‚6991.148409268 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 276548712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74818737 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36264660 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚503493739398144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111083397 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151631436 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6854755948032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7165786796544 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚510348477343232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74818739 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4937438413ÐœÎv 42620 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚562810899599872 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6854755948032 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚510349405330944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚503493739398144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚486913815020096 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6830026322524 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7166009273856 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚486913797980736 trainer/global_step‚5999 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 111084038 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124890906 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚555644866385920 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚493742916328048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚503493721395200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276548712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚503493721395200 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 276522340 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 $ _timestamp‚1.788502933366975e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36265299 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124890906 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151631436 _runtime‚6991.148409268 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 276548712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74818737 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36264660 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚503493739398144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111083397 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151631436 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6854755948032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7165786796544 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚510348477343232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74818739 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚493743841342620 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚562810899599872 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6854755948032 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚510349405330944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚503493739398144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚486913815020096 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6830026322524 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615ê¬@ 2Ú- _wandbruntime‚6991  _runtime‚6991k5WÛÑ 0loss/FRt3/assignment_loss‚1.2222297694001878 1loss/SRqqt1/assignment_loss‚1.732711412757635 2loss/SRqqt4/assignment_loss‚1.8818717883980793 /loss/FRt3/detection_loss‚1.7175335633967603 % _timestamp‚1.7885029333733292e+09 _runtime‚6991.150651548 0loss/FRt4/assignment_loss‚3.5397892203824273 trainer/global_step‚5999 _step‚491 0loss/FBt1/assignment_loss‚0.2280345174377244 1loss/FBt2/assignment_loss‚0.16095675101678353 0loss/FBt3/assignment_loss‚0.2041432081993343 1loss/SRqqt2/detection_loss‚2.1119136112076897 /loss/FBt1/detection_loss‚0.7314200202252824 /loss/FBt2/detection_loss‚2.5715636014938354 0loss/FRt2/assignment_loss‚2.9313632671425984 /loss/FRt1/detection_loss‚2.5464473388840676 .loss/FRt2/detection_loss‚2.499373883735843 1loss/SRqqt3/detection_loss‚0.8997771100712424 /loss/FRt1/assignment_loss‚2.343259324287546 1loss/SRqqt1/detection_loss‚2.0411215364933013 .loss/FBt3/detection_loss‚2.480058971378538 epoch‚4 2loss/SRqqt2/assignment_loss‚1.6231558361223766 -loss/FRt4/detection_loss‚3.15281159302284 /loss/SRqqt4/detection_loss‚2.44565131612446 %loss/total_loss‚42.64309262206993 2loss/SRqqt3/assignment_loss‚0.9068512108349124 0loss/FBt4/assignment_loss‚0.2658100016449359 .loss/FBt4/detection_loss‚2.405243768411524˜ GY\ÜÖ 2loss/SRqqt3/assignment_loss‚0.9068512108349124 0loss/FBt4/assignment_loss‚0.2658100016449359 .loss/FBt4/detection_loss‚2.405243768411524 0loss/FRt3/assignment_loss‚1.2222297694001878 1loss/SRqqt1/assignment_loss‚1.732711412757635 2loss/SRqqt4/assignment_loss‚1.8818717883980793 /loss/FRt3/detection_loss‚1.7175335633967603 % _timestamp‚1.7885029333733292e+09 _runtime‚6991.150651548 0loss/FRt4/assignment_loss‚3.5397892203824273 trainer/global_step‚5999 _step‚491 0loss/FBt1/assignment_loss‚0.2280345174377244 1loss/FBt2/assignment_loss‚0.16095675101678353 0loss/FBt3/assignment_loss‚0.2041432081993343 1loss/SRqqt2/detection_loss‚2.1119136112076897 /loss/FBt1/detection_loss‚0.7314200202252824 /loss/FBt2/detection_loss‚2.5715636014938354 0loss/FRt2/assignment_loss‚2.9313632671425984 /loss/FRt1/detection_loss‚2.5464473388840676 .loss/FRt2/detection_loss‚2.499373883735843 1loss/SRqqt3/detection_loss‚0.8997771100712424 /loss/FRt1/assignment_loss‚2.343259324287546 1loss/SRqqt1/detection_loss‚2.0411215364933013 .loss/FBt3/detection_loss‚2.480058971378538 epoch‚4 2loss/SRqqt2/assignment_loss‚1.6231558361223766 -loss/FRt4/detection_loss‚3.15281159302284 /loss/SRqqt4/detection_loss‚2.44565131612446 %loss/total_loss‚42.64309262206993ë½Z';`Ý‚(:Õ £ÇéÔ“¬n disk.sdb1.in‚0 disk.sdb1.out‚0! disk.dm-0.out‚103479.98046875 cpu‚25.147534832763373 disk.rbd2.in‚0 disk.sda2.in‚3.546875# disk.nvme1n1.out‚34475.83984375 disk.md0.out‚ 749.828125$ memory_percent‚91.98846817016602$ disk./.usageGB‚2710.647186279297 disk.sda.in‚3.546875 disk.nvme0n1.in‚ 939.5390625! disk.nvme0n1.out‚ 69004.140625 disk.rbd0.in‚0 disk.rbd0.out‚ 1.68359375 disk.md0p1.in‚ 4.1796875 disk.sdb.out‚ 779.458984375 disk.md0.in‚ 4.1796875 network.sent‚ 1.5071733e+07' proc.memory.availableMB‚ 1312.609375 disk.sda1.in‚0 disk.dm-0.in‚ 1683.0703125 disk.nvme1n1.in‚ 743.53125# proc.memory.rssMB‚ 3045.43359375 proc.cpu.threads‚17) disk./.usagePercent‚75.79082982235305 disk.rbd2.out‚ 1.908203125 disk.sdb2.out‚ 779.458984375 disk.sda.out‚ 779.458984375 disk.md0p1.out‚ 749.828125 disk.sdb.in‚ 0.6328125 disk.sda2.out‚ 779.458984375 network.recv‚ 2.286684e+06 disk.sda1.out‚0* proc.memory.percent‚18.587851524353027 disk.sdb2.in‚ 0.6328125Õáê“Þ‚(:ˆ £ÇéÔ®™µn gpu.0.gpu‚22.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.157' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.78925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09#/mß‚(:â ²ÇéԮѦm disk.nvme1n1.in‚ 743.53125 disk.sdb2.in‚ 0.6328125) proc.memory.availableMB‚ 1312.61328125 disk.sda.in‚3.546875 disk.sda1.out‚0 disk.md0p1.in‚ 4.1796875 disk.nvme0n1.in‚ 939.5390625 disk.md0.in‚ 4.1796875% disk./.usageGB‚2710.1171073913574 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sdb.out‚780.72021484375 cpu‚25.147263886926982 disk.sda.out‚780.72021484375 disk.sdb.in‚ 0.6328125 disk.md0.out‚ 751.0390625 disk.rbd2.in‚0 proc.cpu.threads‚17 network.sent‚ 1.5156555e+07 disk.dm-0.out‚103761.7265625! disk.sdb2.out‚780.72021484375# memory_percent‚91.9884443283081# disk.nvme0n1.out‚69285.58984375# proc.memory.rssMB‚ 3045.43359375* proc.memory.percent‚18.587851524353027! disk.sda2.out‚780.72021484375# disk.nvme1n1.out‚34476.13671875 disk.rbd2.out‚ 1.908203125) disk./.usagePercent‚75.77600859479101 disk.sda1.in‚0 disk.dm-0.in‚ 1683.0703125 disk.sda2.in‚3.546875 disk.rbd0.in‚0 disk.md0p1.out‚ 751.0390625 network.recv‚ 2.292169e+06 disk.rbd0.out‚ 1.68359375à6• ’à‚(:‡ ²ÇéÔ„ô©m gpu.0.gpu‚4.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.219' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.55475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ʾiá‚(:Þ ÁÇéÔ¥Þ´m disk.sda2.in‚3.546875 disk.sda.out‚782.16064453125! disk.sda2.out‚782.16064453125 disk.nvme0n1.in‚ 939.5390625 disk.md0.in‚ 4.1796875 disk.sdb2.in‚ 0.6328125 disk.sda.in‚3.546875! disk.dm-0.out‚103811.26171875 cpu‚25.146950213148358 disk.sdb.in‚ 0.6328125 disk.rbd2.out‚ 1.908203125 disk.rbd0.out‚ 1.68359375# proc.memory.rssMB‚ 3045.43359375" disk.nvme0n1.out‚ 69334.9453125 disk.rbd2.in‚0 disk.md0p1.out‚ 752.41796875* proc.memory.percent‚18.587851524353027 disk.sdb.out‚782.16064453125# disk.nvme1n1.out‚34476.31640625 disk.rbd0.in‚0$ memory_percent‚91.98837280273438 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb1.in‚0 disk.md0p1.in‚ 4.1796875 disk.md0.out‚ 752.41796875$ disk./.usageGB‚2710.116813659668 disk.nvme1n1.in‚ 743.53125 disk.dm-0.in‚ 1683.0703125) disk./.usagePercent‚75.77600038193003 disk.sdb1.out‚0$ proc.memory.availableMB‚1312.625 network.sent‚ 1.516184e+07 network.recv‚ 2.295486e+06! disk.sdb2.out‚782.16064453125³Qæâ‚(:… ÁÇéÔÙËm gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚126.22' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôÏ4wã‚(:… ÐÇéÔ±•°m gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.168' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x-Ï^ä‚(:Ó ÐÇéÔùÛÉm disk.md0.out‚ 754.40625! disk.sdb2.out‚784.22412109375 disk.md0p1.out‚ 754.40625 disk.nvme0n1.in‚ 939.5390625 disk.nvme1n1.out‚ 34476.53125$ memory_percent‚91.98710918426514) proc.memory.availableMB‚ 1312.83203125 disk.sda1.in‚0 network.sent‚ 1.5170043e+07$ disk./.usageGB‚2710.168025970459 disk.sdb.in‚ 0.6328125 disk.sda2.in‚3.546875 disk.sdb2.in‚ 0.6328125 disk.sda.in‚3.546875* proc.memory.percent‚18.587851524353027 disk.nvme0n1.out‚ 69492.875 disk.rbd0.in‚0 disk.rbd0.out‚ 1.68359375 disk.sdb1.in‚0 disk.sdb1.out‚0 cpu‚25.15112483299727 disk.dm-0.in‚ 1683.0703125 disk.dm-0.out‚ 103969.40625! disk.sda2.out‚784.22412109375 disk.nvme1n1.in‚ 743.53125 proc.cpu.threads‚17 network.recv‚ 2.300265e+06) disk./.usagePercent‚75.77743229957376 disk.md0.in‚ 4.1796875 disk.md0p1.in‚ 4.1796875# proc.memory.rssMB‚ 3045.43359375 disk.rbd2.in‚0 disk.sda.out‚784.22412109375 disk.sdb.out‚784.22412109375 disk.rbd2.out‚ 1.908203125 disk.sda1.out‚0³}î’å‚(:‡ ßÇéÔå„ãn gpu.0.gpu‚59.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—8Óhjæ‚(:ß ßÇéÔ¤‚žo proc.cpu.threads‚17 disk.sdb1.in‚0$ memory_percent‚91.98713302612305 disk.sda.in‚3.546875 disk.sdb.in‚ 0.6328125 disk.sda2.in‚3.546875 disk.sda2.out‚785.7724609375 disk.sda1.out‚0 disk.md0p1.in‚ 4.1796875" disk.nvme1n1.out‚ 34476.6953125' proc.memory.availableMB‚ 1312.828125# proc.memory.rssMB‚ 3045.43359375# disk.nvme0n1.out‚69730.64453125 disk.nvme1n1.in‚ 743.53125 disk.md0.in‚ 4.1796875 disk.md0.out‚ 755.89453125 disk.sda.out‚785.7724609375 disk.rbd0.in‚0 disk.md0p1.out‚ 755.89453125 disk.dm-0.in‚ 1683.0703125 network.sent‚ 1.5175834e+07 network.recv‚ 2.304325e+06 disk.sdb2.out‚785.7724609375 disk.rbd2.out‚ 1.908203125* proc.memory.percent‚18.587851524353027 cpu‚25.160630151839907% disk./.usageGB‚2710.1472549438477 disk.sda1.in‚0! disk.dm-0.out‚104207.33984375) disk./.usagePercent‚75.77685153297637 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sdb2.in‚ 0.6328125 disk.rbd0.out‚ 1.68359375 disk.sdb.out‚785.7724609375 disk.nvme0n1.in‚ 939.5390625V“ç‚(:ˆ îÇéÔÄáül gpu.0.gpu‚62.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚146.997' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.74925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓŠÞkè‚(:à îÇéÔ¹¹Ým) proc.memory.availableMB‚ 1312.83984375 disk.sda1.in‚0 disk.sdb1.out‚0 cpu‚25.16874663344899 proc.cpu.threads‚17 disk.nvme0n1.in‚ 939.5390625 disk.rbd2.out‚ 1.919921875 disk.sda.out‚787.01416015625 disk.sdb1.in‚0 disk.dm-0.in‚ 1683.0703125 disk.sda2.in‚3.546875 disk.sdb2.in‚ 0.6328125# disk.nvme0n1.out‚69762.85546875 disk.nvme1n1.in‚ 743.53125 network.sent‚ 1.5182176e+07 disk.sda.in‚3.546875 disk.sdb.in‚ 0.6328125 disk.sdb.out‚787.01416015625* proc.memory.percent‚18.587851524353027) disk./.usagePercent‚75.77860055238209 disk.sda1.out‚0 network.recv‚ 2.309473e+06" disk.nvme1n1.out‚ 34477.1015625 disk.md0.out‚ 757.078125# proc.memory.rssMB‚ 3045.43359375% disk./.usageGB‚2710.2098083496094 disk.md0p1.out‚ 757.078125 disk.md0.in‚ 4.1796875 disk.rbd2.in‚0! disk.sda2.out‚787.01416015625 disk.rbd0.in‚0 disk.rbd0.out‚ 1.69921875 disk.md0p1.in‚ 4.1796875! disk.dm-0.out‚104239.95703125! disk.sdb2.out‚787.01416015625$ memory_percent‚91.98706150054932¼åÖ³œé‚(:‘ ýÇéÔÌ ­m gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R­]ê‚(:Ò ýÇéÔº÷Þm cpu‚25.178397556469058) disk./.usagePercent‚75.78385358358909 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.dm-0.in‚ 1683.0703125! disk.dm-0.out‚104694.94140625 disk.nvme1n1.in‚ 743.53125# disk.nvme1n1.out‚34477.30078125 disk.md0.in‚ 4.1796875 disk.sdb.in‚ 0.6328125 disk.md0p1.out‚ 758.99609375 disk.sda2.out‚ 788.9921875 disk.rbd2.out‚ 1.982421875 disk.sdb2.out‚ 788.9921875' proc.memory.availableMB‚ 1312.828125 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.sdb.out‚ 788.9921875 disk.nvme0n1.in‚ 939.5390625 disk.sdb2.in‚ 0.6328125 disk.sda.in‚3.546875 disk.sda1.out‚0# proc.memory.rssMB‚ 3045.43359375 network.sent‚ 1.5187474e+07 network.recv‚ 2.312871e+06 disk.md0.out‚ 758.99609375* proc.memory.percent‚18.587851524353027 disk.rbd0.out‚ 1.69921875 disk.md0p1.in‚ 4.1796875! disk.nvme0n1.out‚ 70217.640625 disk.rbd2.in‚0 disk.sda.out‚ 788.9921875% disk./.usageGB‚2710.3976821899414 disk.sda2.in‚3.546875 disk.sda1.in‚0$ memory_percent‚91.98713302612305¡|82ë- _wandbruntime‚7102  _runtime‚7102‡³ß= ìÝT SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 112616637 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚36786437 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 7242929552896 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 279744457 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14401142784 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚569705086714880 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _step‚492 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 701042176 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 698106808 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 681067448 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚570 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚499844039570954 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated[àô’Y‚492940485928592 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1185 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 153188029 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189768192 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚516754840969216 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 279744457 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3725 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20711 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚492940468889232 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚570 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚516754840969216 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 13002869760 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 683039232 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 6903570681722 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 279765170 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 14135738368 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 6929476268544 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚509825364700672 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 112616868 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14401142784 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚615 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚509825364700672 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 126556430 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1268 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3725 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 6929476268544 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚516755542011392 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 6930159307776 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 153188029 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚231 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚229 trainer/global_step‚6044 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1185kú0:h*íâT QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚570 % _timestamp‚1.7885030449319568e+09 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 6930159307776 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 279765170 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚75830202 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚569704883884544 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 409073152 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13684416521 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1187728393 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚509825382703616 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚516755542011392 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1185 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚44 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 683039232 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 126556428 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 153208740 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32006 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚509825382703616 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚562462157161984 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 _runtime‚7102.709972425 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 701042176 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14401142784 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 153208740 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 126556430 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚36786666 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14401142784 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚569705086714880 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _step‚492 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 701042176 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 698106808 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 681067448 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚570 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚499844039570954 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚492940485928592 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1185 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 153188029 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189768192 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚516754840969216 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 279744457 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3725 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20711 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚492940468889232 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚570 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚516754840969216 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 13002869760 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 683039232 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 6903570681722 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 279765170 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 14135738368 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 6929476268544 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚509825364700672 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 112616868 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14401142784 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚615 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚509825364700672 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 126556430 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1268 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3725 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 6929476268544 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚516755542011392 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 6930159307776 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 153188029 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚231 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚229 trainer/global_step‚6044 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1185 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 112616637 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚36786437 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 7242929552896 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 279744457 FDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 126571906 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚562518400775680 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 6904739960054 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 701042176 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚230 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 202830336 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1187728393 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚509879625607680 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚516810292769280 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚44 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 6904058892606 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚232 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 409073152 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 681067448 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14401142784 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚509879643610624 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚562518424715776 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 112631437 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚36791675 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚569761641289216 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13205766144 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚570 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 126571906 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 6929966119424 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 6930649158656 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 153196843 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚499894953887022 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 279789460 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135738368 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 683039232 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 279768747 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 126571906 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚562518400775680 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 6904739960054 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 701042176 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚230 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 202830336 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1187728393 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚509879625607680 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚516810292769280 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚44 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 6904058892606 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚232 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 409073152 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 681067448 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14401142784 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚509879643610624 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚562518424715776 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 112631437 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚36791675 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚569761641289216 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13205766144 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚570 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 126571906 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 6929966119424 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 6930649158656 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 153196843 XDDeviceStatsMonitor.on_validation_batch_end/requesˆ(—Eõ ted_bytes.all.freed‚499894953887022 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 279789460 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135738368 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 683039232 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 279768747 B&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.25412848591804504 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 *jet/accuracy_1_of_2‚0.7541725635528564 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40123456716537476 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3809368312358856 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991071224212646 ,jet/accuracy_4_of_4‚0.008314451202750206 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_7‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5301183462142944 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9736111164093018 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5532187223434448 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.49382713437080383 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5502896904945374 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5822916030883789 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14444445073604584 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33712777495384216 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4227251410484314 ,"Purity/0FRt0SRqqt3FBt/event_purity‚0.125 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15695224702358246 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151560306549 jet/accuracy_1_of_6‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.016631312668323517 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2095286101102829 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11264487355947495 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.936170220375061 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.014150943607091904 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004031242337077856 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.145417720079422 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.36065274477005005 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5240790247917175 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.40909090638160706 +jet/accuracy_2_of_5‚0.38264212012290955 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9210526347160339 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9911637902259827 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14245374500751495 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0590277798473835 particle/accuracy_2_of_3‚1 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.719298243522644 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.977011501789093 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_4‚1 jet/accuracy_6_of_6‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.0941520407795906 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_6‚0.5512820482254028 particle/accuracy_3_of_6‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.37799566984176636 *"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.5 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12786364555358887 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5172972679138184 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5754589438438416 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.13063064217567444 ,jet/accuracy_4_of_5‚0.010394265875220299 particle/accuracy_3_of_5‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_5‚1 'particle/f_score‚0.4009665548801422 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9929367899894714 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6086478233337402 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9926664233207703 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.02964537963271141 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4486589729785919 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6354166865348816 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18818631768226624 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05267857387661934 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5115593075752258 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5036380290985107 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5892857313156128 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14725631475448608 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_3_of_4‚0.0677100121974945 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.019607843831181526 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24791347980499268 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5172972679138184 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.973958432674408 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.40697672963142395 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5474398136138916 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25632181763648987 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3083333373069763 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 +jet/accuracy_4_of_6‚0.01923076994717121 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15695224702358246 )loss/val_total_loss‚42.13696317368463 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5186530351638794 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.013957307673990726 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999850988388 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_7‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5748109817504883 +jet/accuracy_3_of_6‚0.11538461595773697 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5099999904632568 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.029743069782853127 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4796936810016632 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9978967308998108 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3757499158382416 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1905864179134369 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.25833335518836975 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9629628658294678 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1464632898569107 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3644675016403198 *jet/accuracy_2_of_2‚0.1056903675198555 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9931343197822571 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1410336196422577 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3679579794406891 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9917359948158264 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 particle/accuracy_6_of_6‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5388956069946289 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4270833134651184 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.637993335723877 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9973360300064087 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2589285671710968 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3819224536418915 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21221314370632172 +"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.75 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 _runtime‚7119.675526725 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2730768024921417 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9444443583488464 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5106959342956543 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.020671172067523003 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3854166865348816 particle/accuracy_2_of_7‚1 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.411465585231781 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22359472513198853 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2386854887008667 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333432674408 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.10805453360080719 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7348485589027405 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4298798441886902 +jet/accuracy_1_of_1‚0.48989665508270264 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15227742493152618 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5760167241096497 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_5‚0.07125575840473175 particle/accuracy_2_of_6‚1 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5126723051071167 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 *jet/accuracy_1_of_4‚0.9149512052536011 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.24166664481163025 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2500697374343872 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.38117560744285583 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07272971421480179 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2560141086578369 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5538428425788879 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 jet/accuracy_5_of_7‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40371671319007874 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9922142028808594 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.003448275849223137 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5606903433799744 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1888095885515213 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5258620381355286 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10283688455820084 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 *"Purity/1FRt*SRqqt4FBt/event_purity‚0.5 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1350574642419815 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.527742326259613 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5144969820976257 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.024097545072436333 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.×àH¬‡'2585969865322113 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.812688946723938 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5230869650840759 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.31854358315467834 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09516157954931259 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4185185730457306 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.11112909018993378 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08751136809587479 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0009920635493472219 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4287387728691101 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5737591981887817 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9895833134651184 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1841573566198349 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3790178596973419 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5258620381355286 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.41714879870414734 ;"Purity/2FRt0SRqqt2FBt/event_purity‚0.032894738018512726 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1184210553765297 jet/accuracy_3_of_7‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14510062336921692 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21372589468955994 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1888892501592636 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.006627116817981005 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9775537848472595 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4205387234687805 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.018493322655558586 <"Purity/*FRt3SRqqt0FBt/event_purity‚0.0023041474632918835 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6174702048301697 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.48521339893341064 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 jet/accuracy_2_of_7‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5395362973213196 _step‚494 particle/accuracy_1_of_6‚1 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5625 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.487651526927948 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5333333015441895 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 8"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.535304605960846 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3672351837158203 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333432674408 trainer/global_step‚6044 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.48945924639701843 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0012570245889946818 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16892965137958527 *particle/sensitivity‚0.323812335729599 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 epoch‚4 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.09656982123851776 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9905831813812256 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20687788724899292 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1031746044754982 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03480057045817375 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5507451891899109 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5740384459495544 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 *particle/specificity‚0.946226954460144 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1585574895143509 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5402805209159851 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16107723116874695 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1400996893644333 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5047992467880249 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09157437831163406 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8772574067115784 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5331600308418274 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 % _timestamp‚1.7885030618961835e+09 <"Purity/*FRt3SRqqt*FBt/event_purity‚0.0023041474632918835 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.10802468657493591 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2553471326828003 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4128933250904083 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3472222089767456 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.329291969537735 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5259058475494385 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5077437162399292 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036º!<‚kXõ¾¦ ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5380228161811829 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9920586347579956 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01632670685648918 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 -jet/accuracy_5_of_5‚0.0026881720405071974 particle/accuracy_1_of_1‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.11551723629236221 particle/accuracy_1_of_2‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9479167461395264 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15127746760845184 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.35625001788139343 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.13131313025951385 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4739326238632202 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 *"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26123711466789246 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 *jet/accuracy_1_of_5‚0.9324885606765747 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4962165355682373 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6849933862686157 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13057802617549896 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9937641620635986 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4178919494152069 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.49295639991760254 *jet/accuracy_2_of_3‚0.2331918627023697 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9924412369728088 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12450326979160309 + Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4375 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_4‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3472222089767456 -"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5625 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.019607843831181526 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4546668529510498 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99875807762146 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7857142686843872 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9846134185791016 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13728207349777222 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.25412848591804504 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 *jet/accuracy_1_of_2‚0.7541725635528564 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40123456716537476 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3809368312358856 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991071224212646 ,jet/accuracy_4_of_4‚0.008314451202750206 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_7‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5301183462142944 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9736111164093018 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5532187223434448 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.49382713437080383 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5502896904945374 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5822916030883789 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14444445073604584 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33712777495384216 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4227251410484314 ,"Purity/0FRt0SRqqt3FBt/event_purity‚0.125 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15695224702358246 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151560306549 jet/accuracy_1_of_6‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.016631312668323517 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2095286101102829 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11264487355947495 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.936170220375061 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.014150943607091904 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004031242337077856 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.145417720079422 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.36065274477005005 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5240790247917175 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.40909090638160706 +jet/accuracy_2_of_5‚0.38264212012290955 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9210526347160339 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9911637902259827 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14245374500751495 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0590277798473835 particle/accuracy_2_of_3‚1 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.719298243522644 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.977011501789093 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_4‚1 jet/accuracy_6_of_6‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.0941520407795906 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_6‚0.5512820482254028 particle/accuracy_3_of_6‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.37799566984176636 *"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.5 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12786364555358887 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5172972679138184 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5754589438438416 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.13063064217567444 ,jet/accuracy_4_of_5‚0.010394265875220299 particle/accuracy_3_of_5‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_5‚1 'particle/f_score‚0.4009665548801422 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9929367899894714 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6086478233337402 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9926664233207703 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.02964537963271141 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4486589729785919 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6354166865348816 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18818631768226624 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05267857387661934 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5115593075752258 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5036380290985107 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5892857313156128 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14725631475448608 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_3_of_4‚0.0677100121974945 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.019607843831181526 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24791347980499268 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5172972679138184 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.973958432674408 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.40697672963142395 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5474398136138916 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25632181763648987 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3083333373069763 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 +jet/accuracy_4_of_6‚0.01923076994717121 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15695224702358246 )loss/val_total_loss‚42.13696317368463 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5186530351638794 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.013957307673990726 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999850988388 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_7‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5748109817504883 +jet/accuracy_3_of_6‚0.11538461595773697 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5099999904632568 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.029743069782853127 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4796936810016632 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9978967308998108 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3757499158382416 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1905864179134369 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.25833335518836975 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9629628658294678 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1464632898569107 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3644675016403198 *jet/accuracy_2_of_2‚0.1056903675198555 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9931343197822571 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1410336196422577 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3679579794406891 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9917359948158264 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 particle/accuracy_6_of_6‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5388956069946289 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4270833134651184 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.637993335723877 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9973360300064087 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2589285671710968 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3819224536418915 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21221314370632172 +"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.75 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 _runtime‚7119.675526725 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2730768024921417 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9444443583488464 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5106959342956543 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.020671172067523003 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3854166865348816 particle/accuracy_2_of_7‚1 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.411465585231781 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22359472513198853 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2386854887008667 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333432674408 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.10805453360080719 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7348485589027405 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4298798441886902 +jet/accuracy_1_of_1‚0.48989665508270264 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15227742493152618 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5760167241096497 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_5‚0.07125575840473175 particle/accuracy_2_of_6‚1 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5126723051071167 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 *jet/accuracy_1_of_4‚0.9149512052536011 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.24166664481163025 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2500697374343872 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.38117560744285583 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07272971421480179 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2560141086578369 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5538428425788879 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 jet/accuracy_5_of_7‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40371671319007874 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9922[ÃÉÚ:142028808594 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.003448275849223137 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5606903433799744 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1888095885515213 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5258620381355286 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10283688455820084 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 *"Purity/1FRt*SRqqt4FBt/event_purity‚0.5 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1350574642419815 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.527742326259613 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5144969820976257 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.024097545072436333 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2585969865322113 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.812688946723938 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5230869650840759 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.31854358315467834 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09516157954931259 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4185185730457306 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.11112909018993378 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08751136809587479 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0009920635493472219 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4287387728691101 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5737591981887817 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9895833134651184 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1841573566198349 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3790178596973419 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5258620381355286 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.41714879870414734 ;"Purity/2FRt0SRqqt2FBt/event_purity‚0.032894738018512726 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1184210553765297 jet/accuracy_3_of_7‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14510062336921692 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21372589468955994 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1888892501592636 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.006627116817981005 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9775537848472595 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4205387234687805 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.018493322655558586 <"Purity/*FRt3SRqqt0FBt/event_purity‚0.0023041474632918835 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6174702048301697 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.48521339893341064 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 jet/accuracy_2_of_7‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5395362973213196 _step‚494 particle/accuracy_1_of_6‚1 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5625 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.487651526927948 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5333333015441895 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 8"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.535304605960846 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3672351837158203 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333432674408 trainer/global_step‚6044 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.48945924639701843 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0012570245889946818 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16892965137958527 *particle/sensitivity‚0.323812335729599 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 epoch‚4 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.09656982123851776 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9905831813812256 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20687788724899292 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1031746044754982 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03480057045817375 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5507451891899109 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5740384459495544 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 *particle/specificity‚0.946226954460144 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1585574895143509 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5402805209159851 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16107723116874695 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1400996893644333 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5047992467880249 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09157437831163406 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8772574067115784 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5331600308418274 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 % _timestamp‚1.7885030618961835e+09 <"Purity/*FRt3SRqqt*FBt/event_purity‚0.0023041474632918835 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.10802468657493591 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2553471326828003 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4128933250904083 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3472222089767456 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.329291969537735 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5259058475494385 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5077437162399292 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.990845799446106 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.45048731565475464 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.029069768264889717 particle/accuracy_1_of_7‚1 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.194996178150177 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4677681624889374 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15802384912967682 6validation_average_jet_accuracy‚0.4368738830089569 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9937959909439087 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5357270240783691 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4870056211948395 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13150790333747864 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0202985517680645 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 jet/accuracy_1_of_7‚1 particle/accuracy_5_of_7‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18644756078720093 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_5_of_5‚1 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9980475902557373 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.007709750905632973 (particle/accuracy‚0.8490409851074219 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_7_of_7‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9883729219436646 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07288049161434174 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_7‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9900230169296265î$Kˆdö‚(:Ù ›ÈéÔµúÚl disk.sdb1.in‚0 disk.md0p1.out‚ 762.20703125! disk.dm-0.out‚105185.79296875 disk.rbd2.in‚0 disk.sdb2.out‚ 792.330078125# memory_percent‚92.2060489654541 cpu‚25.19293573905841 network.sent‚ 1.5298539e+07 disk.sdb.in‚ 0.6328125 disk.nvme0n1.in‚ 939.5390625# disk.nvme0n1.out‚70708.02734375 disk.sda1.out‚0* proc.memory.percent‚22.593355178833008 disk.nvme1n1.in‚ 743.53125) disk./.usagePercent‚75.79272688657737% disk./.usageGB‚2710.7150344848633 disk.md0p1.in‚ 4.1796875( proc.memory.availableMB‚ 1276.9609375" proc.memory.rssMB‚ 3701.6953125 disk.sda2.in‚3.546875! disk.nvme1n1.out‚ 34477.765625 disk.rbd2.out‚ 2.0615234375 disk.sda.out‚ 792.330078125 disk.rbd0.out‚ 1.76171875 proc.cpu.threads‚17 disk.dm-0.in‚ 1683.0703125 disk.sda1.in‚0 disk.rbd0.in‚0 network.recv‚ 2.322622e+06 disk.sdb1.out‚0 disk.sdb.out‚ 792.330078125 disk.sda2.out‚ 792.330078125 disk.sdb2.in‚ 0.6328125 disk.sda.in‚3.546875 disk.md0.in‚ 4.1796875 disk.md0.out‚ 762.20703125Èåµý÷‚(:„ ›ÈéÔ§ªíl gpu.0.gpu‚7.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚32.0 gpu.0.powerWatts‚86.17' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚21.5425 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§™2ø- _wandbruntime‚7129  _runtime‚7129æ±3ÇùÁ _runtime‚7129.744790934 _step‚495 & lr-AdamW/pg1‚0.0006169480160638725 & lr-AdamW/pg2‚0.0006169480160638725 trainer/global_step‚6049 % _timestamp‚1.7885030719675195e+09!Î4ÌúÆ & lr-AdamW/pg1‚0.0006169480160638725 & lr-AdamW/pg2‚0.0006169480160638725 trainer/global_step‚6049 % _timestamp‚1.7885030719675195e+09 _runtime‚7129.744790934 _step‚495ïÓê‘Ó2û- _wandbruntime‚7129  _runtime‚7129íîyHû'üõO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76012328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚500877157462196 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 280262458 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚510911493742080 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6940599508992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚510911493742080 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚573 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36880923 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚493962521779344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126853732 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚563610805732864 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 trainer/global_step‚6049 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 153408726 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚493962504739984 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚517852093251072 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153435070 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126853734 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36881496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6915560697216 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 153408726 $ _timestamp‚1.788503071968358e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6941509480448 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚575 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚563610781792768 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153435070 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6941509480448 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 112893251 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚563610805732864 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 trainer/global_step‚6049 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 153408726 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚493962504739984 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚517852093251072 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153435070 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126853734 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36881496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6915560697216 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 153408726 $ _timestamp‚1.788503071968358e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6941509480448 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚575 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚563610781792768 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153435070 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185ðÀ”› 2þ- _wandbruntime‚7130  _runtime‚7130vqÇ•'ÿ€N S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚500957292330018 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280307449 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153433557 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7255602314752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚510993130830848 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126873894 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 % _timestamp‚1.7885030729416096e+09 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6941720969216 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6941720969216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚494041522334528 _step‚497 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126873892 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 112910893 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36886500 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚494041539373888 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 153459927 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36887079 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6916677970702 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280333821 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280307449 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚6049 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 153433557 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚563701007497216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1,s '€…N H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚581 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚510993130830848 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6942630953984 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7255828986368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚517935761784832 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚517934833797120 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76024395 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚579 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚6049 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 153433557 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚563701007497216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚500957292330018 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280307449 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153433557 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7255602314752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚510993130830848 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126873894 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 % _timestamp‚1.7885030729416096e+09 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6941720969216 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6941720969216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚494041522334528 _step‚497 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126873892 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 112910893 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36886500 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚494041539373888 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 153459927 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36887079 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6916677970702 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280333821 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280307449 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚575434556605440 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6971437952324 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚521989134967296 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚514992466036224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚568122090955264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚497913401506544 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 113780736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚575434817716224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 113780134 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76615580 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76615578 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37164556 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7312489590272 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6997596905472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6996686934016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚497913384467184 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 282541706 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚602 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚504884839458868 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚504883914444504 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 154679972 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6997596905472 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7312726760960 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127861732 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚514992466036224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 127861734 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚521990062941696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 282515360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚521990062941696 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237170688 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144…Û (úO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚568122067015168 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚600 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 154679972 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6996686934016 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127861732 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 trainer/global_step‚6099 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 154653628 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚521989134967296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127861732 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚514992466036224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 127861734 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚521990062941696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 282515360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚521990062941696 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237170688 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 _runtime‚7181.686814748 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚575434556605440 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6971437952324 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚521989134967296 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚514992466036224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚568122090955264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚497913401506544 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 113780736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚575434817716224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 113780134 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76615580 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76615578 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37164556 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7312489590272 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6997596905472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6996686934016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚497913384467184 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 282541706 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚602 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚504884839458868 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚504883914444504 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 154679972 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6997596905472 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7312726760960 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚515074067119104 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 282560351 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚615 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚515074085122048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚613 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚568212292719616 trainer/global_step‚6099 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154704829 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚522072803501056 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚515074085122048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6997808394240 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7313659296256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚515074067119104 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 282586723 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37170187 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 282560351 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 282586723 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚504964974326898 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚497992402061728 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37170800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154704829 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚575525952015872 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚504964049312326 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚497992419101088 _step‚501 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113797830 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6972555225810 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154678459 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚522071875513344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚575526202627584 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚568212316659712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚522072803501056 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154704829 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚522072803501056 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚515074085122048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6997808394240 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7313659296256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonit€Ëj#õor.on_train_batch_end/allocated_bytes.large_pool.freed‚515074067119104 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 282586723 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37170187 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 282560351 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 282586723 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚504964974326898 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚497992402061728 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37170800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154704829 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚575525952015872 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚504964049312326 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚497992419101088 _step‚501 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113797830 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6972555225810 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154678459 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚522071875513344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚575526202627584 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚568212316659712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚522072803501056 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚515074067119104 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615õì Ü42“- _wandbruntime‚7182  _runtime‚7182õIÜZ”Ô 0loss/FRt4/assignment_loss‚3.3095403361320495 2loss/SRqqt3/assignment_loss‚1.0133157138381954 /loss/FBt1/detection_loss‚0.5758201118827944 trainer/global_step‚6099 /loss/FRt1/detection_loss‚1.7498564887917547 /loss/FRt2/detection_loss‚2.0017185846033194 .loss/FBt2/detection_loss‚2.442545461654663 %loss/total_loss‚41.44286314348314 0loss/FRt1/assignment_loss‚1.9059606329961256 0loss/FBt3/assignment_loss‚0.2277050055689974 1loss/SRqqt2/detection_loss‚2.1260793926539243 _step‚502 0loss/FRt2/assignment_loss‚3.1231455331047377 1loss/FBt1/assignment_loss‚0.05818251466354994 1loss/FBt4/assignment_loss‚0.17160384150058514 0loss/SRqqt1/detection_loss‚2.351930409669876 0loss/FRt3/assignment_loss‚2.0353452137538364 2loss/SRqqt2/assignment_loss‚1.5783364905251398 2loss/SRqqt4/assignment_loss‚1.5050931146575346 _runtime‚7182.649318182 0loss/FBt2/assignment_loss‚0.2242496978186864 /loss/FRt3/detection_loss‚2.1024839794263244 0loss/SRqqt4/detection_loss‚2.719712491388674 epoch‚5 % _timestamp‚1.7885031248723469e+09 1loss/SRqqt1/assignment_loss‚1.151362243067959 .loss/FRt4/detection_loss‚3.161171452999115 1loss/SRqqt3/detection_loss‚1.1329176265258933 .loss/FBt3/detection_loss‚2.392021675904592 .loss/FBt4/detection_loss‚2.382765130354808¼'±_•Ù /loss/FRt1/detection_loss‚1.7498564887917547 /loss/FRt2/detection_loss‚2.0017185846033194 .loss/FBt2/detection_loss‚2.442545461654663 %loss/total_loss‚41.44286314348314 0loss/FRt1/assignment_loss‚1.9059606329961256 0loss/FBt3/assignment_loss‚0.2277050055689974 1loss/SRqqt2/detection_loss‚2.1260793926539243 _step‚502 0loss/FRt2/assignment_loss‚3.1231455331047377 1loss/FBt1/assignment_loss‚0.05818251466354994 1loss/FBt4/assignment_loss‚0.17160384150058514 0loss/SRqqt1/detection_loss‚2.351930409669876 0loss/FRt3/assignment_loss‚2.0353452137538364 2loss/SRqqt2/assignment_loss‚1.5783364905251398 2loss/SRqqt4/assignment_loss‚1.5050931146575346 _runtime‚7182.649318182 0loss/FBt2/assignment_loss‚0.2242496978186864 /loss/FRt3/detection_loss‚2.1024839794263244 0loss/SRqqt4/detection_loss‚2.719712491388674 epoch‚5 % _timestamp‚1.7885031248723469e+09 1loss/SRqqt1/assignment_loss‚1.151362243067959 .loss/FRt4/detection_loss‚3.161171452999115 1loss/SRqqt3/detection_loss‚1.1329176265258933 .loss/FBt3/detection_loss‚2.392021675904592 .loss/FBt4/detection_loss‚2.382765130354808 0loss/FRt4/assignment_loss‚3.3095403361320495 2loss/SRqqt3/assignment_loss‚1.0133157138381954 /loss/FBt1/detection_loss‚0.5758201118827944 trainer/global_step‚6099ö€Mç‘–‚(:† ×ÈéÔ‰ÍËl gpu.0.gpu‚10.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚180.196' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.049 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜŒ`Ba—‚(:Ö ×ÈéÔìäm disk.md0p1.out‚ 768.90234375 disk.sdb.out‚799.27392578125) proc.memory.availableMB‚ 2056.95703125 disk.sdb1.out‚0 disk.rbd2.out‚2.1875 disk.sda.out‚799.27392578125) proc.memory.percent‚18.06325912475586 network.sent‚ 1.5544741e+07 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sdb.in‚ 0.6328125 disk.nvme0n1.in‚ 939.5390625 disk.nvme0n1.out‚ 73044.71875! proc.memory.rssMB‚ 2959.484375 disk.md0.in‚ 4.1796875 disk.md0.out‚ 768.90234375 disk.sdb2.in‚ 0.6328125 disk.sdb1.in‚0 disk.rbd2.in‚0( disk./.usagePercent‚75.7794817816985 disk.rbd0.out‚ 1.912109375! disk.sdb2.out‚799.27392578125$ memory_percent‚87.44533061981201 cpu‚25.191396641148184 disk.dm-0.out‚ 107743.859375 network.recv‚ 2.349022e+06$ disk./.usageGB‚2710.241325378418 disk.dm-0.in‚ 1683.0703125 disk.sda2.in‚3.546875 disk.nvme1n1.in‚ 743.53125 proc.cpu.threads‚17 disk.sda1.out‚0 disk.md0p1.in‚ 4.1796875! disk.sda2.out‚799.27392578125! disk.nvme1n1.out‚ 34699.140625 disk.sda.in‚3.546875”1ø“˜‚(:ˆ æÈéÔ¢Ó¬k gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚178.629' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.65725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F"!¿\™‚(:Ñ æÈéÔµ‹àl disk.sda.out‚ 801.369140625 disk.sda1.in‚0 disk.sdb1.in‚0$ memory_percent‚87.44754791259766 disk.sdb.in‚ 0.6328125 disk.sdb.out‚ 801.369140625 disk.nvme0n1.in‚ 939.5390625 disk.rbd2.in‚0 disk.sda1.out‚0 disk.rbd0.out‚ 1.97509765625 disk.sdb1.out‚0 disk.rbd2.out‚2.1875 disk.sdb2.out‚ 801.369140625) proc.memory.percent‚18.06325912475586) disk./.usagePercent‚75.77569021310262 disk.md0p1.in‚ 4.1796875 disk.md0p1.out‚ 770.93359375 disk.sda2.out‚ 801.369140625 disk.sda2.in‚3.546875 disk.sda.in‚3.546875 cpu‚25.191001953074526 proc.cpu.threads‚17 disk.nvme1n1.out‚ 34712.09375 disk.rbd0.in‚0 network.sent‚ 1.5652681e+07% disk./.usageGB‚2710.1057205200195 disk.dm-0.out‚107757.6796875" disk.nvme0n1.out‚ 73045.5859375 disk.md0.in‚ 4.1796875& proc.memory.availableMB‚ 2056.59375! proc.memory.rssMB‚ 2959.484375 network.recv‚ 2.354687e+06 disk.dm-0.in‚ 1683.0703125 disk.nvme1n1.in‚ 743.53125 disk.md0.out‚ 770.93359375 disk.sdb2.in‚ 0.6328125ÖU–J“š‚(:ˆ õÈéÔ¬¤¡m gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚135.517' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.87925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽÔ„Î\›‚(:Ñ õÈéÔ¢ÊÚn disk.rbd0.in‚0 disk.md0p1.in‚ 4.1796875) proc.memory.percent‚18.06325912475586 disk.md0.in‚ 4.1796875 disk.nvme1n1.in‚ 743.53125 disk.rbd2.in‚0 disk.sda.in‚3.546875$ memory_percent‚87.44606971740723! proc.memory.rssMB‚ 2959.484375 cpu‚25.190592020069804 disk.sda2.out‚ 802.85546875 disk.md0.out‚ 772.35546875 disk.rbd2.out‚2.1875 disk.sdb2.out‚ 802.85546875 disk.md0p1.out‚ 772.35546875 proc.cpu.threads‚17# disk.nvme1n1.out‚34712.26171875 network.sent‚ 1.5660119e+07 disk.sdb.in‚ 0.6328125 disk.sdb.out‚ 802.85546875 disk.nvme0n1.in‚ 939.5390625 disk.dm-0.in‚ 1683.0703125( disk./.usagePercent‚75.7768516396369 disk.sda1.out‚0 disk.rbd0.out‚ 1.97509765625 disk.sdb1.in‚0 disk.sdb1.out‚0( proc.memory.availableMB‚ 2056.8359375 network.recv‚ 2.357853e+06 disk.sda2.in‚3.546875$ disk./.usageGB‚2710.147258758545 disk.sdb2.in‚ 0.6328125 disk.dm-0.out‚107885.2578125# disk.nvme0n1.out‚73176.24609375 disk.sda.out‚ 802.85546875 disk.sda1.in‚0@‘Á“œ‚(:ˆ „ÉéÔäþèk gpu.0.gpu‚31.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.461' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.11525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c§xˆ\‚(:Ñ „ÉéÔ²í¸m" disk.nvme0n1.out‚ 73472.5390625 disk.nvme1n1.in‚ 743.53125" disk.nvme1n1.out‚ 34712.5546875 disk.md0.in‚ 4.1796875! disk.sdb2.out‚804.27294921875 disk.dm-0.out‚ 108185.09375) proc.memory.percent‚18.06325912475586 network.sent‚ 1.5663852e+07 disk.sda1.in‚0! disk.sda2.out‚804.27294921875 disk.rbd2.out‚2.1875 disk.rbd0.out‚ 1.97509765625 disk.sdb1.out‚0 disk.sda.in‚3.546875 disk.sda1.out‚0 network.recv‚ 2.362127e+06 disk.sdb.out‚804.27294921875& proc.memory.availableMB‚ 2056.59375 cpu‚25.190276099830054 proc.cpu.threads‚17( disk./.usagePercent‚75.7803551181355 disk.nvme0n1.in‚ 939.5390625 disk.sdb2.in‚ 0.6328125 disk.sdb1.in‚0! proc.memory.rssMB‚ 2959.484375 disk.sdb.in‚ 0.6328125 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1683.0703125$ memory_percent‚87.44754791259766 disk.sda2.in‚3.546875 disk.sda.out‚804.27294921875 disk.md0p1.in‚ 4.1796875 disk.md0p1.out‚ 773.71875 disk.md0.out‚ 773.71875$ disk./.usageGB‚2710.272560119629Š–³2ž- _wandbruntime‚7235  _runtime‚7235μ÷ÆŸÀ & lr-AdamW/pg1‚0.0006019481014359341 & lr-AdamW/pg2‚0.0006019481014359341 trainer/global_step‚6149 $ _timestamp‚1.788503177408272e+09 _runtime‚7235.185725039 _step‚503ÑɔˠŠ& lr-AdamW/pg1‚0.0006019481014359341 & lr-AdamW/pg2‚0.0006019481014359341 trainer/global_step‚6149 $ _timestamp‚1.788503177408272e+09 _runtime‚7235.185725039 _step‚503÷:Dªz2¡- _wandbruntime‚7235  _runtime‚7235hÙ$¹û'¢õO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7052774359040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77218828 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚601 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚526126176683520 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114666275 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚501864264194384 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚526126176683520 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 114666878 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚526127104657920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128869732 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155924874 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128869732 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 $ _timestamp‚1.788503177409492e+09 _step‚504 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218296320 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37447447 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7371018042880 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚508890671426812 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77218830 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7053684330496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚526127104657920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7052774359040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚580004394220544 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 284768262 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 284794608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 trainer/global_step‚6149 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 _runtime‚7235.187232086 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚603 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚508891596441176 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7026407232428 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7053684330496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155898530 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128869734 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚519073420327424 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚519073420327424 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128869734 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚519073402324480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7027315207432 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚572633352237568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0ÙÈ"€ó£úO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 $ _timestamp‚1.788503177409492e+09 _step‚504 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218296320 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37447447 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7371018042880 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚508890671426812 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77218830 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7053684330496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚526127104657920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7052774359040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚580004394220544 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 284768262 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 284794608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 trainer/global_step‚6149 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 _runtime‚7235.187232086 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚603 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚508891596441176 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7026407232428 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7053684330496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155898530 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128869734 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚519073420327424 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚519073420327424 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128869734 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚519073402324480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7027315207432 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚572633352237568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7052774359040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77218828 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚601 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_bG²±ñ atch_start/segment.all.peak‚1185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚526126176683520 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114666275 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚501864264194384 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚526126176683520 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 114666878 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚526127104657920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128869732 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155924874 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128869732 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44øÖÁ¡“2¤- _wandbruntime‚7236  _runtime‚7236¢¾¯'¥€N UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 155949731 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚580095807234560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚501943281788928 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 284813253 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚526209845217280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 284813253 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚580095560817152 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37453771 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114684666 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7028432480918 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚526209845217280 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 % _timestamp‚1.7885031785168955e+09 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7054805804032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7053895819264 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚572723601882112 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚508970806294634 trainer/global_step‚6149 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 155949731 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284839625 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7371982875136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚508971731309206 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7053895819264 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _step‚505 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37453153 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7372205352448 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114684046¬ÒÚ" '¦…N TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 155949731 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚580095807234560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚501943281788928 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 284813253 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚526209845217280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 284813253 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚580095560817152 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37453771 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114684666 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7028432480918 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚526209845217280 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 % _timestamp‚1.7885031785168955e+09 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7054805804032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7053895819264 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚572723601882112 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚508970806294634 trainer/global_step‚6149 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 155949731 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284839625 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7371982875136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚508971731309206 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7053895819264 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _step‚505 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37453153 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7372205352448 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114684046 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760ù¢¶8Ù2§- _wandbruntime‚7236  _runtime‚7236fµU¨Ï _runtime‚7236.297778715 /loss/FRt4/detection_loss‚2.8503818273544312 /loss/FBt2/detection_loss‚2.2334834795731764 /loss/FBt3/detection_loss‚1.4234361615445879 0loss/FBt4/assignment_loss‚0.4005718219382556 /loss/FRt2/assignment_loss‚3.234600009237017 2loss/SRqqt2/assignment_loss‚1.4840361208400943 /loss/FBt4/detection_loss‚2.5391772634842815 _step‚506 epoch‚5 0loss/FBt1/assignment_loss‚0.2051392074325934 0loss/FBt3/assignment_loss‚0.3147607626807256 /loss/FRt2/detection_loss‚2.6474074008209363 .loss/FRt1/assignment_loss‚3.20693916636844 0loss/FRt3/assignment_loss‚1.6178998901293828 2loss/SRqqt3/assignment_loss‚1.0748162687724006 1loss/SRqqt1/detection_loss‚2.0274684476852416 /loss/SRqqt2/detection_loss‚2.49858297272162 /loss/FRt1/detection_loss‚2.2496617265782715 /loss/FRt4/assignment_loss‚3.692116723457972 1loss/SRqqt4/assignment_loss‚1.413354151855622 .loss/FRt3/detection_loss‚1.619364393903659 $ _timestamp‚1.788503178520641e+09 .loss/FBt1/detection_loss‚0.667267271325857 trainer/global_step‚6149 1loss/SRqqt3/detection_loss‚0.9411709699137457 1loss/SRqqt4/detection_loss‚2.2491267289434163 &loss/total_loss‚42.482353913847646 0loss/FBt2/assignment_loss‚0.5169614508273438 1loss/SRqqt1/assignment_loss‚1.374629696458578=:øwZ©Ô 0loss/FBt1/assignment_loss‚0.2051392074325934 0loss/FBt3/assignment_loss‚0.3147607626807256 /loss/FRt2/detection_loss‚2.6474074008209363 .loss/FRt1/assignment_loss‚3.20693916636844 0loss/FRt3/assignment_loss‚1.6178998901293828 2loss/SRqqt3/assignment_loss‚1.0748162687724006 1loss/SRqqt1/detection_loss‚2.0274684476852416 /loss/SRqqt2/detection_loss‚2.49858297272162 /loss/FRt1/detection_loss‚2.2496617265782715 /loss/FRt4/assignment_loss‚3.692116723457972 1loss/SRqqt4/assignment_loss‚1.413354151855622 .loss/FRt3/detection_loss‚1.619364393903659 $ _timestamp‚1.788503178520641e+09 .loss/FBt1/detection_loss‚0.667267271325857 trainer/global_step‚6149 1loss/SRqqt3/detection_loss‚0.9411709699137457 1loss/SRqqt4/detection_loss‚2.2491267289434163 &loss/total_loss‚42.482353913847646 0loss/FBt2/assignment_loss‚0.5169614508273438 1loss/SRqqt1/assignment_loss‚1.374629696458578 _runtime‚7236.297778715 /loss/FRt4/detection_loss‚2.8503818273544312 /loss/FBt2/detection_loss‚2.2334834795731764 /loss/FBt3/detection_loss‚1.4234361615445879 0loss/FBt4/assignment_loss‚0.4005718219382556 /loss/FRt2/assignment_loss‚3.234600009237017 2loss/SRqqt2/assignment_loss‚1.4840361208400943 /loss/FBt4/detection_loss‚2.5391772634842815 _step‚506 epoch‚5ú¶—䑪‚(:† “ÉéÔ‹éàl gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚175.512' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆÃl»a«‚(:Ö “ÉéÔ®¿ m disk.sda.in‚3.546875 disk.rbd0.in‚0 cpu‚25.18991921816776 disk.sda1.in‚0 disk.rbd0.out‚ 1.97509765625 disk.sdb1.in‚0 disk.sdb.out‚ 806.318359375 disk.md0.in‚ 4.1796875 disk.sdb2.out‚ 806.318359375 disk.sda.out‚ 806.318359375 disk.md0p1.in‚ 4.1796875 disk.dm-0.in‚ 1683.0703125! disk.dm-0.out‚108477.25390625 disk.sdb.in‚ 0.6328125 disk.md0p1.out‚ 775.69921875 disk.sda2.in‚3.546875! proc.memory.rssMB‚ 2959.484375) proc.memory.percent‚18.06325912475586 proc.cpu.threads‚17" disk.nvme1n1.out‚ 34712.7578125 disk.md0.out‚ 775.69921875 disk.rbd2.in‚0 disk.sdb2.in‚ 0.6328125 network.sent‚ 1.5669748e+07 network.recv‚ 2.366001e+06# disk.nvme0n1.out‚73764.49609375 disk.rbd2.out‚2.1875 disk.sda1.out‚0 disk.sda2.out‚ 806.318359375$ memory_percent‚87.44587898254395) disk./.usagePercent‚75.78385369024963% disk./.usageGB‚2710.3976860046387 disk.nvme0n1.in‚ 939.5390625 disk.sdb1.out‚0( proc.memory.availableMB‚ 2056.8671875 disk.nvme1n1.in‚ 743.53125*ÚØ¡‘¬‚(:† ¢ÉéÔ¸àØk gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚179.856' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Mf0f­‚(:Û ¢ÉéÔ Þñl disk.nvme1n1.in‚ 743.53125# disk.nvme1n1.out‚34829.37109375 disk.rbd2.out‚2.1875 disk.md0p1.in‚ 4.1796875 cpu‚25.18959949854953 network.recv‚ 2.371409e+06 disk.sdb.out‚807.81982421875 disk.nvme0n1.in‚ 939.5390625 disk.rbd2.in‚0 disk.sdb2.in‚ 0.6328125 disk.sda1.out‚0 disk.dm-0.in‚ 1683.0703125 disk.md0.out‚ 777.1328125! disk.sdb2.out‚807.81982421875 disk.sda.in‚3.546875! proc.memory.rssMB‚ 2959.484375 proc.cpu.threads‚17 disk.sda2.in‚3.546875 disk.sda1.in‚0) proc.memory.availableMB‚ 2056.75390625 disk.sdb1.out‚0) disk./.usagePercent‚75.77609765633532 disk.rbd0.out‚ 1.97509765625$ memory_percent‚87.44657039642334) proc.memory.percent‚18.06325912475586$ disk./.usageGB‚2710.120292663574 disk.sdb.in‚ 0.6328125! disk.nvme0n1.out‚ 73927.078125 disk.md0.in‚ 4.1796875 disk.sda.out‚807.81982421875 disk.rbd0.in‚0 network.sent‚ 1.5777038e+07! disk.sda2.out‚807.81982421875 disk.sdb1.in‚0 disk.md0p1.out‚ 777.1328125! disk.dm-0.out‚108756.44921875{KG›’®‚(:‡ ±ÉéÔ¹Á˜m gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚182.641' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.66025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>gñ×f¯‚(:Û ±ÉéÔø¾ôm disk.rbd0.out‚ 1.97509765625 disk.md0p1.out‚ 778.5078125! disk.dm-0.out‚109027.22265625# disk.nvme1n1.out‚34829.52734375 disk.rbd2.out‚2.1875 disk.sda.in‚3.546875 disk.sda.out‚809.24462890625 network.sent‚ 1.5785635e+07 disk.sdb.in‚ 0.6328125 disk.nvme1n1.in‚ 744.32421875$ memory_percent‚91.45965576171875) proc.memory.percent‚18.06325912475586 proc.cpu.threads‚17 disk.rbd0.in‚0# proc.memory.availableMB‚1399.25 network.recv‚ 2.381286e+06 disk.sda2.in‚3.546875! disk.sda2.out‚809.24462890625 disk.md0p1.in‚ 4.1796875) disk./.usagePercent‚75.77500342593629$ disk./.usageGB‚2710.081157684326" disk.nvme0n1.out‚ 74197.6953125 disk.rbd2.in‚0! disk.sdb2.out‚809.24462890625 disk.sda1.in‚0 disk.nvme0n1.in‚ 939.5390625 disk.sda1.out‚0 disk.sdb1.in‚0! proc.memory.rssMB‚ 2959.484375 disk.dm-0.in‚ 1683.86328125 disk.sdb.out‚809.24462890625 disk.md0.in‚ 4.1796875 disk.md0.out‚ 778.5078125 disk.sdb2.in‚ 0.6328125 disk.sdb1.out‚0 cpu‚25.189202535482078ªò•ô2°- _wandbruntime‚7287  _runtime‚7287ðëq¬S±Á & lr-AdamW/pg1‚0.0005944236758105071 & lr-AdamW/pg2‚0.0005944236758105!ÜDÕt071 trainer/global_step‚6199 % _timestamp‚1.7885032301885211e+09 _runtime‚7287.965761798 _step‚507Ÿì˜Ì²Æ & lr-AdamW/pg1‚0.0005944236758105071 & lr-AdamW/pg2‚0.0005944236758105071 trainer/global_step‚6199 % _timestamp‚1.7885032301885211e+09 _runtime‚7287.965761798 _step‚507ûÞ… ¾2³- _wandbruntime‚7287  _runtime‚7287à /ú'´ôO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 115555578 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚530264146374144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 287021164 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77822080 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129877732 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚6199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚621 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157143432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚505815143921584 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77822078 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚512898353423484 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7108861784064 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7108861784064 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚512897428409120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚508 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚619 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚584573649001984 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚523154374618624 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 129877734 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚530263218399744 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚523154356615680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 115554957 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚523154356615680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚530264146374144 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7082284487536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚577144637459968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7429238226944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7429011542016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7083192462540®Scÿ'µùO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 287021164 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77822080 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129877732 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚6199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚621 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157143432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚505815143921584 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77822078 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚512898353423484 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7108861784064 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7108861784064 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚512897428409120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚508 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚619 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚584573649001984 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚523154374618624 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 129877734 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚530263218399744 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚523154356615680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 115554957 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚523154356615680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚530264146374144 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7082284487536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚577144637459968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7429238226944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7429011542016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7083192462540 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 115555578 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚530264146374144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0üòq‡2¶- _wandbruntime‚7288  _runtime‚7288wýúé'·þM SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129897894 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 129897892 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚577234863164416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7430409562112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 157168263 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7109983244288 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 157168263 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115572645 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚530345958945792 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115573276 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 129897892 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚530346886933504 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚523235975701504 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚505894161516128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 287092527 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 157194633 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77834145 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚577234887104512 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287092527 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37738502 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚584665296666624 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157194633 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚631 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7084309736026 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129897894 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚509 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚523235993704448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚629 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 287066155 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77834143 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7083401760814 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚523235993704448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7110893229056 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚584665048152064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚512978488291514 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192&c,™'¸ƒN VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37739131 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129897894 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 129897892 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚577234863164416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7430409562112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 157168263 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7109983244288 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 157168263 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115572645 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 S?DeviceStatsMonitor.on_train_batch_end/aá]«7âllocated_bytes.all.freed‚530345958945792 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115573276 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 129897892 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚530346886933504 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚523235975701504 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚505894161516128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 287092527 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 157194633 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77834145 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚577234887104512 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287092527 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37738502 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚584665296666624 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157194633 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚631 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7084309736026 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129897894 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚509 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚523235993704448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚629 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 287066155 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77834143 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7083401760814 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚523235993704448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7110893229056 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚584665048152064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚512978488291514 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚516904185391428 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚527235328909824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7165859180544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78425328 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚512 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚534400260115968 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚534400260115968 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130885734 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7487610381312 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚534400260115968 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130885734 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7487610381312 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚516904185391428 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚527235328909824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7165859180544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78425328 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚512€ôªz 2Ê- _wandbruntime‚7341  _runtime‚7341çšgñ'Ë€N E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚625 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116461733 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7139279015922 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7166980654080 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7166070669312 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚527316947995648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7166980654080 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38024340 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 158439535 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130905892 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 289345429 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚534483928649728 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚589234693345280 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7488769532928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚6249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚527316947995648 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚516984320259250 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 116462360 _runtime‚7341.525987397 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130905894 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚581746172326912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7140186991134 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚513 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚527316929992704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚509845058282688 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289345429 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158413165 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 289319057 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38024965 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚509845041243328 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130905892 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚627 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78437395 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚516984320259250 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 116462360 _runtime‚7341.525987397 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130905894 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚581746172326912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7140186991134 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚513 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚527316929992704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚509845058282688 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289345429 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158413165 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 289319057 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38024965 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚509845041243328 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130905892 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚627 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78437395 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116461733 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7139279015922 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7166980654080 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7166070669312 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚527316947995648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7166980654080 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38024340 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 158439535 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130905892 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 289345429 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚534483928649728 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚589234693345280k2Í- _wandbruntime‚7341  _runtime‚7341J£$\ÎÖ /loss/FBt3/detection_loss‚2.1474220126867296 /loss/FBt1/detection_loss‚0.6407888401332508 %loss/total_loss‚41.88446546890248 0loss/FRt2/assignment_loss‚2.8116460094849267 2loss/SRqqt1/assignment_loss‚1.4605850158259273 2loss/SRqqt4/assignment_loss‚1.9988634511828423 0loss/FBt3/assignment_loss‚0.1511625374194409 _step‚514 0loss/FRt1/assignment_loss‚2.8161474883556368 1loss/FBt4/assignment_loss‚0.18053110105773323 1loss/SRqqt2/detection_loss‚2.6445395400126777 /loss/FRt1/detection_loss‚1.7769577423248848 0loss/SRqqt4/detection_loss‚2.221924026166239 % _timestamp‚1.7885032837504501e+09 0loss/SRqqt2/assignment_loss‚1.48321542601722 2loss/SRqqt3/assignment_loss‚1.0476862366478412 /loss/FRt2/detection_loss‚2.5713613610714674 .loss/FRt4/detection_loss‚2.969480512663722 1loss/SRqqt3/detection_loss‚1.0114777418506196 _runtime‚7341.527511804 trainer/global_step‚6249 /loss/FRt4/assignment_loss‚3.054067488759756 1loss/FBt2/assignment_loss‚0.21174783601866995 /loss/FRt3/detection_loss‚1.5562841686001723 1loss/SRqqt1/detection_loss‚1.7926557185128331 0loss/FRt3/assignment_loss‚2.1589077495146487 1loss/FBt1/assignment_loss‚0.15930242990715104 .loss/FBt2/detection_loss‚2.236280643939972 /loss/FBt4/detection_loss‚2.7814303907481106 epoch‚5Öfá£aÏÛ 2loss/SRqqt4/assignment_loss‚1.9988634511828423 0loss/FBt3/assignment_loss‚0.1511625374194409 _step‚514 0loss/FRt1/assignment_loss‚2.8161474883556368 1loss/FBt4/assignment_loss‚0.18053110105773323 1loss/SRqqt2/detection_loss‚2.6445395400126777 /loss/FRt1/detection_loss‚1.7769577423248848 0loss/SRqqt4/detection_loss‚2.221924026166239 % _timestamp‚1.7885032837504501e+09 0loss/SRqqt2/assignment_loss‚1.48321542601722 2loss/SRqqt3/assignment_loss‚1.0476862366478412 /loss/FRt2/detection_loss‚2.5713613610714674 .loss/FRt4/detection_loss‚2.969480512663722 1loss/SRqqt3/detection_loss‚1.0114777418506196 _runtime‚7341.527511804 trainer/global_step‚6249 /loss/FRt4/assignment_loss‚3.054067488759756 1loss/FBt2/assignment_loss‚0.21174783601866995 /loss/FRt3/detection_loss‚1.5562841686001723 1loss/SRqqt1/detection_loss‚1.7926557185128331 0loss/FRt3/assignment_loss‚2.1589077495146487 1loss/FBt1/assignment_loss‚0.15930242990715104 .loss/FBt2/detection_loss‚2.236280643939972 /loss/FBt4/detection_loss‚2.7814303907481106 epoch‚5 /loss/FBt3/detection_loss‚2.1474220126867296 /loss/FBt1/detection_loss‚0.6407888401332508 %loss/total_loss‚41.88446546890248 0loss/FRt2/assignment_loss‚2.8116460094849267 2loss/SRqqt1/assignment_loss‚1.4605850158259273‚†‘–ÓЂ(:ˆ üÉéÔÜ„Òk gpu.0.gpu‚25.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚182.981' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.74525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HŠýeÑ‚(:Ú üÉéÔ„Ôíl network.recv‚ 2.40464e+06 disk.sda.in‚3.546875 disk.sda.out‚817.8623046875) disk./.usagePercent‚75.76955296608435 disk.md0.out‚ 786.8046875 disk.sdb1.in‚0 disk.sdb.out‚817.8623046875 disk.sda2.in‚3.546875! disk.nvme1n1.out‚ 34830.953125 disk.rbd2.in‚0 disk.sda1.in‚0$ memory_percent‚91.46015644073486 cpu‚25.187489413334077 disk.rbd2.out‚2.1875 disk.sdb2.in‚ 0.6328125 network.sent‚ 1.5917996e+07 disk.sdb1.out‚0 disk.nvme1n1.in‚ 744.32421875% disk./.usageGB‚2709.8862228393555 disk.md0p1.in‚ 4.1796875 disk.nvme0n1.in‚ 939.5390625 disk.md0.in‚ 4.1796875! proc.memory.rssMB‚ 2959.484375 disk.rbd0.out‚ 1.97509765625" disk.nvme0n1.out‚ 74679.5703125 disk.sda1.out‚0) proc.memory.availableMB‚ 1399.16796875 disk.md0p1.out‚ 786.8046875 disk.sdb2.out‚817.8623046875 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-0.in‚ 1683.86328125 disk.dm-0.out‚109510.5234375 disk.sdb.in‚ 0.6328125 disk.sda2.out‚817.8623046875) proc.memory.percent‚18.06325912475586™¹…å“Ò‚(:ˆ ‹ÊéÔÚؽk gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚170.891' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.72275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä)‘hlÓ‚(:á ‹ÊéÔý…†m disk.md0.out‚ 788.43359375 disk.sdb2.in‚ 0.6328125 disk.sda.in‚3.546875 disk.md0p1.out‚ 788.43359375! disk.sdb2.out‚819.55029296875 disk.rbd0.in‚0$ memory_percent‚91.46170616149902) proc.memory.percent‚18.06325912475586) disk./.usagePercent‚75.78005134894002 disk.sdb1.in‚0 disk.md0p1.in‚ 4.1796875 disk.sdb.in‚ 0.6328125 disk.sda2.in‚3.546875# disk.nvme0n1.out‚74839.56640625 cpu‚25.187175246495798 proc.cpu.threads‚17 disk.nvme1n1.in‚ 744.32421875 disk.sda.out‚819.55029296875! proc.memory.rssMB‚ 2959.484375! disk.sda2.out‚819.55029296875 disk.rbd2.out‚2.1875 network.sent‚ 1.6025828e+07 network.recv‚ 2.409953e+06 disk.sdb.out‚819.55029296875 disk.rbd2.in‚0 disk.sda1.in‚0% disk./.usageGB‚2710.2616958618164! disk.dm-0.out‚109670.66015625 disk.nvme0n1.in‚ 939.5390625 disk.rbd0.out‚ 1.97509765625( proc.memory.availableMB‚ 1398.9140625 disk.sdb1.out‚0 disk.dm-0.in‚ 1683.86328125 disk.nvme1n1.out‚ 34831.09375 disk.sda1.out‚0 disk.md0.in‚ 4.1796875¡Y•h’Ô‚(:‡ šÊéÔ™§´k gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚127.446' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.8615 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ãtð°ÉÕ‚(:Ü šÊéÔ½ýl network.sent‚ 1.6038571e+07) disk./.usagePercent‚75.78352965555304 disk.sda1.out‚0 network.recv‚ 2.427589e+06 disk.dm-0.in‚ 1683.86328125 disk.sdb2.in‚ 0.6328125# memory_percent‚91.4632797241211& proc.memory.availableMB‚ 1398.65625 disk.sdb.out‚820.96923828125" disk.nvme0n1.out‚ 74977.9140625! proc.memory.rssMB‚ 2959.484375) proc.memory.percent‚18.06325912475586 disk.sda2.in‚3.546875% disk./.usageGB‚2710.3860969543457 disk.rbd0.in‚0 disk.sdb1.in‚0 cpu‚25.186830314729914 proc.cpu.threads‚17 disk.sda.out‚820.96923828125! disk.sda2.out‚820.96923828125 disk.nvme0n1.in‚ 939.5390625" disk.nvme1n1.out‚ 34831.4921875 disk.md0.in‚ 4.1796875 disk.rbd2.out‚2.1875 disk.md0p1.out‚ 789.79296875 disk.dm-0.out‚ 109809.40625 disk.sdb.in‚ 0.6328125 disk.md0.out‚ 789.79296875 disk.rbd2.in‚0! disk.sdb2.out‚820.96923828125 dLMÁžisk.sda.in‚3.546875 disk.sdb1.out‚0 disk.nvme1n1.in‚ 744.32421875 disk.sda1.in‚0 disk.rbd0.out‚ 1.97509765625 disk.md0p1.in‚ 4.1796875¦Jn]2Ö- _wandbruntime‚7393  _runtime‚7393^ÛyMÇ×Á trainer/global_step‚6299 % _timestamp‚1.7885033360022295e+09 _runtime‚7393.779520971 _step‚515 & lr-AdamW/pg1‚0.0005793337280050643 & lr-AdamW/pg2‚0.0005793337280050643ÎØÞÌØÆ _step‚515 & lr-AdamW/pg1‚0.0005793337280050643 & lr-AdamW/pg2‚0.0005793337280050643 trainer/global_step‚6299 % _timestamp‚1.7885033360022295e+09 _runtime‚7393.779520971ƒCæãæ2Ù- _wandbruntime‚7393  _runtime‚7393è8Zhü'ÚöO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7194038997752 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚538538229806592 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 291526968 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38305751 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7221036634112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 291553314 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7545693394944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚520910942373736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚538537301832192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79028578 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚531316265198080 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚586167231844864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚538537301832192 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117334329 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214102016 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚7393.780846607 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 159659580 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 trainer/global_step‚6299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚513716920415344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚531316265198080 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚531316283201024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚513716903375984 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131893734 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238042112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 159633236 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7221946605568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚593712687197696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 131893732 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131893734 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚593712925239808 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚586167207904768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7194946972756 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚538537301832192 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117334329 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214102016 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚7393.780846607 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 159659580 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 trainer/global_step‚6299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚513716920415344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚531316265198080 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚531316283201024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚513716903375984 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131893734 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238042112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 159633236 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7221946605568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚593712687197696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 131893732 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131893734 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚593712925239808 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚586167207904768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7194946972756 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885033370203552e+09 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 117352690 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚538620042378240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7223068079104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79040645 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 291598331 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117352068 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 159684437 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 159658067 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7223068079104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7222158094336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235060224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7196064246242 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7222158094336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38311425 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚586257457549312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚531397902286848 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚593804319451136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7546861901824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 291598331 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 291571959 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚586257433609216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚520992002256130 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚531397902286848 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79040643 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 131913894 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚513795920970528 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _runtime‚7394.798632446 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7546626841600 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚513795938009888 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚538620970365952 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131913892 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 _step‚517 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 291571959 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚531397884283904 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640ÔÿœðÞ…N VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7222158094336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235060224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7196064246242 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7222158094336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38311425 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚586257457549312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚531397902286848 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚5938043194511µÌ!36 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7546861901824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 291598331 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 291571959 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚586257433609216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚520992002256130 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚531397902286848 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79040643 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 131913894 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚513795920970528 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _runtime‚7394.798632446 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7546626841600 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚513795938009888 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚538620970365952 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131913892 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 _step‚517 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 291571959 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚531397884283904 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 159658067 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚520991077241558 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131913892 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚538620970365952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885033370203552e+09 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 117352690 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚538620042378240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7223068079104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79040645 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 291598331 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117352068 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 159684437 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 159658067 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7223068079104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 trainer/global_step‚6349 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79631828 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚524918624370408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _runtime‚ 7450.4013385 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79631830 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚535397237492224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚524917699356044 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38594520 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚542675271522816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132901732 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚542675271522816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚542674343548416 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 118226988 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7885033926238143e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132901732 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7277124059136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 132901734 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160904482 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚598282408964096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 160878138 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 293779870 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚535397237492224 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚535397219489280 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192Àt(©žïùO A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7250824227864 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚517667783103184 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚535397237492224 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚535397219489280 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7277124059136 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7603915836928 _step‚520 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚535397219489280 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132901734 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 293806216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7278034030592 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚638 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚590678517067264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚517667800142544 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 trainer/global_step‚6349 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79631828 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.a§àŠ÷allocated‚524918624370408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _runtime‚ 7450.4013385 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79631830 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚535397237492224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚524917699356044 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38594520 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚542675271522816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132901732 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚542675271522816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚542674343548416 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 118226988 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7885033926238143e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132901732 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7277124059136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 132901734 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160904482 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚598282408964096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 160878138 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 293779870 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7278245519360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38600879 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚524998759238438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _runtime‚7451.422121283 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚542758012082176 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚535478856578048 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚535478856578048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚535478838575104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 263194624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚590768718831616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 293824861 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7605069285888 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 # _timestamp‚1.78850339364298e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7278245519360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 239254528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚521 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 293824861 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 293851233 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118244774 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118244125 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38600232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚649 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚598374051312128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚598373788117504 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚517746800697728 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 trainer/global_step‚6349 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚517746817737088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7279155504128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 293851233 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚542757084094464 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7279155504128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79643895 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132921894 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640Îf>Ö 'òƒN SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132921894 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7605308540416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚590768742771712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132921892 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160902969 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 160929339 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚524997834223866 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚535478838575104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚647 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7251033526138 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 160929339 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118244125 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38600232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚649 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚598374051312128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚598373788117504 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚517746800697728 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 trainer/global_step‚6349 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚517746817737088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7279155504128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 293851233 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚542757084094464 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7279155504128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79643895 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132921894 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚542758012082176 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7278245519360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38600879 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚524998759238438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _runtime‚7451.422121283 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚542758012082176 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚535478856578048 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚535478856578048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚535478838575104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 263194624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚590768718831616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 293824861 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7605069285888 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 # _timestamp‚1.78850339364298e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7278245519360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 239254528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚521 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 293824861 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 293851233 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118244774 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚546811385264640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚539478173780480 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7333211484160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80235080 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7662057384960 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 _step‚524 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38883485 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7662288264192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚539478191783424 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚528924456338352 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 trainer/global_step‚6399 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚546811385264640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚546812313239040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚539478191783424 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚539478191783424 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚528924456338352 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 trainer/global_step‚6399 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚546811385264640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚546812313239040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚539478191783424 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚546811385264640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚539478173780480 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760ŒŒÕ˜ñ2„- _wandbruntime‚7508  _runtime‚7508Œ„ÁÇ%…€N B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚539559810869248 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7334332944384 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚602943478705152 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133929894 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119135631 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚634 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 296077763 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7307818756458 _step‚525 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚636 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚595280004054016 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 162147871 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 239254528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚546894125810688 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 trainer/global_step‚6399 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 162147871 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7663450711040 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚529005516220746 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚529004591206174 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚595280027994112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚521697697464288 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885034507919128e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7335242929152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133929894 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚539559792866304 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 296104135 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80247143 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚539559810869248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 263194624 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7334332944384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚7508.570248474 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚546895053798400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38888488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 296077763 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 119136267 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 162174241 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7335242929152 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133929892 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7663211456512 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38889122 XDDeviceStatsMonitor.on_train_batch_end/`•áø?inactive_split_bytes.all.freed‚602943215510528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144½w '†…N S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚529004591206174 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚595280027994112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚521697697464288 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885034507919128e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7335242929152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133929894 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚539559792866304 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 296104135 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80247143 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚539559810869248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 263194624 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7334332944384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚7508.570248474 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚546895053798400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38888488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 296077763 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 119136267 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 162174241 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7335242929152 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133929892 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7663211456512 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38889122 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚602943215510528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚539559810869248 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7334332944384 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚602943478705152 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133929894 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119135631 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚634 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 296077763 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7307818756458 _step‚525 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚636 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚595280004054016 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 162147871 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 239254528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚546894125810688 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 trainer/global_step‚6399 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 162147871 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7663450711040 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚529005516220746 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944ø¢»2‡- _wandbruntime‚7508  _runtime‚7508ëÕ&VˆÐ 1loss/SRqqt4/detection_loss‚2.4632156435400248 /loss/FBt2/detection_loss‚2.8903472453355787 .loss/FBt4/detection_loss‚2.622231046358744 $ _timestamp‚1.788503450794786e+09 /loss/FBt3/assignment_loss‚0.169980535236391 /loss/FBt3/detection_loss‚2.3067208727200827 epoch‚5 1loss/FBt4/assignment_loss‚0.28248969782495503 .loss/FBt1/detection_loss‚0.739658031558645 %loss/total_loss‚44.18728621959615 /loss/FRt2/assignment_loss‚3.163778219085473 0loss/FBt1/assignment_loss‚0.1017802740132248 .loss/FRt1/detection_loss‚2.175086093243649 /loss/FRt4/detection_loss‚3.2771183997392654 1loss/SRqqt1/detection_loss‚2.0534862684352055 .loss/FRt3/detection_loss‚1.916653703769733 1loss/SRqqt3/detection_loss‚1.1329219287547927 2loss/SRqqt1/assignment_loss‚1.3427927632416998 2loss/SRqqt3/assignment_loss‚0.8957364199987541 _runtime‚7508.571782132 trainer/global_step‚6399 .loss/FRt3/assignment_loss‚2.15743867487743 1loss/FBt2/assignment_loss‚0.07721744675654918 .loss/FRt2/detection_loss‚2.320195556582453 1loss/SRqqt2/detection_loss‚2.7411462582390884 _step‚526 /loss/FRt1/assignment_loss‚2.289068276231939 0loss/FRt4/assignment_loss‚3.5246827065944673 2loss/SRqqt2/assignment_loss‚1.7183554727455665 2loss/SRqqt4/assignment_loss‚1.8251846847124398YÞÇ[‰Õ trainer/global_step‚6399 .loss/FRt3/assignment_loss‚2.15743867487743 1loss/FBt2/assignment_loss‚0.07721744675654918 .loss/FRt2/detection_loss‚2.320195556582453 1loss/SRqqt2/detection_loss‚2.7411462582390884 _step‚526 /loss/FRt1/assignment_loss‚2.289068276231939 0loss/FRt4/assignment_loss‚3.5246827065944673 2loss/SRqqt2/assignment_loss‚1.7183554727455665 2loss/SRqqt4/assignment_loss‚1.8251846847124398 1loss/SRqqt4/detection_loss‚2.4632156435400248 /loss/FBt2/detection_loss‚2.8903472453355787 .loss/FBt4/detection_loss‚2.622231046358744 $ _timestamp‚1.788503450794786e+09 /loss/FBt3/assignment_loss‚0.169980535236391 /loss/FBt3/detection_loss‚2.3067208727200827 epoch‚5 1loss/FBt4/assignment_loss‚0.28248969782495503 .loss/FBt1/detection_loss‚0.739658031558645 %loss/total_loss‚44.18728621959615 /loss/FRt2/assignment_loss‚3.163778219085473 0loss/FBt1/assignment_loss‚0.1017802740132248 .loss/FRt1/detection_loss‚2.175086093243649 /loss/FRt4/detection_loss‚3.2771183997392654 1loss/SRqqt1/detection_loss‚2.0534862684352055 .loss/FRt3/detection_loss‚1.916653703769733 1loss/SRqqt3/detection_loss‚1.1329219287547927 2loss/SRqqt1/assignment_loss‚1.3427927632416998 2loss/SRqqt3/assignment_loss‚0.8957364199987541 _runtime‚7508.571782132ŽΆv‘Š‚(:† ¡ËéÔýö”n gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚143.872' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.968 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vT_f‹‚(:Û ¡ËéÔ™õ­n disk.rbd0.in‚0 disk.nvme0n1.in‚ 939.5390625 disk.md0.out‚ 804.64453125$ memory_percent‚91.46122932434082 proc.cpu.threads‚17 disk.rbd2.out‚2.1875 disk.sda2.in‚3.546875( proc.memory.availableMB‚ 1398.9921875 network.recv‚ 2.492838e+06 disk.sda.in‚3.546875 disk.md0p1.out‚ 804.64453125 disk.sdb.out‚836.43798828125# disk.nvme0n1.out‚76621.09765625# disk.nvme1n1.out‚35092.51953125 disk.rbd2.in‚0 disk.sdb2.in‚ 0.6328125 disk.sda1.out‚0 disk.md0.in‚ 4.1796875! disk.sdb2.out‚836.43798828125 disk.dm-0.in‚ 1683.86328125 disk.dm-0.out‚111713.6171875! proc.memory.rssMB‚ 2959.484375 cpu‚25.18384038270568 disk.rbd0.out‚2.1875 disk.sdb1.in‚0 disk.md0p1.in‚ 4.1796875 disk.sdb.in‚ 0.6328125! disk.sda2.out‚836.43798828125 disk.nvme1n1.in‚ 744.32421875 network.sent‚ 1.6320093e+07% disk./.usageGB‚2710.8856658935547 disk.sdb1.out‚0) proc.memory.percent‚18.06325912475586) disk./.usagePercent‚75.79749781217916 disk.sda.out‚836.43798828125 disk.sda1.in‚0«,£ç‘Œ‚(:† °ËéÔ‰„çl gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚175.512' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Áª\Z‚(:Ï °ËéÔÍØÒm disk.md0p1.out‚ 806.109375 disk.sdb.out‚ 837.951171875 disk.nvme0n1.in‚ 939.5390625 disk.rbd2.out‚2.1875 disk.sdb2.in‚ 0.6328125 disk.sda.out‚ 837.951171875 disk.rbd0.out‚2.1875# disk.nvme0n1.out‚76988.08984375 disk.md0.in‚ 4.1796875) proc.memory.availableMB‚ 1398.99609375( disk./.usagePercent‚75.8045158618717 disk.nvme1n1.in‚ 744.32421875! disk.nvme1n1.out‚ 35092.765625 disk.md0.out‚ 806.109375% disk./.usageGB‚2711.1366653442383 disk.sdb1.in‚0 disk.sda2.in‚3.546875 disk.sdb2.out‚ 837.951171875 disk.md0p1.in‚ 4.1796875! disk.dm-0.out‚112080.85546875$ memory_percent‚91.46120548248291 proc.cpu.threads‚17 network.recv‚ 2.498207e+06 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.sda2.out‚ 837.951171875 disk.sda.in‚3.546875 disk.sda1.out‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 1683.86328125 disk.sdb.in‚ 0.6328125 cpu‚25.183546526162413 network.sent‚ 1.6416999e+07 disk.sda1.in‚0! proc.memory.rssMB‚ 2959.484375) proc.memory.percent‚18.06325912475586Ò¿éøŽ‚(:’ ¿ËéÔÁëºm gpu.0.gpu‚24.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚127.106' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚31.776499999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z@›é`‚(:Õ ¿ËéÔûŒÔm disk.sdb.in‚ 0.6328125 disk.md0.out‚ 807.5703125$ memory_percent‚91.46122932434082 cpu‚25.183215823835518! disk.nvme0n1.out‚ 77145.328125" disk.nvme1n1.out‚ 35239.4453125 disk.md0p1.in‚ 4.1796875 disk.dm-0.out‚112384.7734375( proc.memory.availableMB‚ 1398.9921875! proc.memory.rssMB‚ 2959.484375 disk.sdb1.in‚0 disk.md0p1.out‚ 807.5703125 disk.nvme1n1.in‚ 744.32421875 disk.md0.in‚ 4.1796875 disk.rbd2.out‚2.1875 disk.sda.out‚839.47119140625 disk.sda1.in‚0 disk.rbd0.out‚2.1875 disk.dm-0.in‚ 1683.86328125) proc.memory.percent‚18.06325912475586% disk./.usageGB‚2711.1355323791504 disk.sda2.in‚3.546875 disk.sdb2.in‚ 0.6328125! disk.sdb2.out‚839.47119140625 disk.sda1.out‚0 network.recv‚ 2.502089e+06 disk.sda.in‚3.546875 disk.sdb1.out‚0! disk.sda2.out‚839.47119140625 disk.rbd2.in‚0 network.sent‚ 1.64206e+07) disk./.usagePercent‚75.80448418369366 disk.sdb.out‚839.47119140625 disk.nvme0n1.in‚ 939.5390625 disk.rbd0.in‚0 proc.cpu.threads‚17®LµÏ“‚(:ˆ ÎËéÔç‚›m gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚123.435' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.85875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 y.[‘‚(:Ð ÎËéÔäµÊm disk.sda1.out‚0 disk.sdb1.out‚0 disk.rbd2.out‚2.1875 disk.rbd0.out‚2.1875 disk.md0p1.out‚ 809.59765625 disk.sda2.in‚3.546875 disk.nvme1n1.in‚ 744.32421875 disk.sda1.in‚0 disk.sdb1.in‚0 disk.dm-0.in‚ 1683.86328125 disk.sdb.out‚ 841.58203125! proc.memory.rssMB‚ 2959.484375 proc.cpu.threads‚17 disk.nvme0n1.in‚ 939.5390625# disk.nvme1n1.out‚35386.25390625 network.recv‚ 2.506583e+06% disk./.usageGB‚2711.1355323791504 disk.sda.out‚ 841.58203125! disk.dm-0.out‚112741.30078125! disk.nvme0n1.out‚ 77355.046875$ memory_percent‚91.46134853363037 disk.md0.in‚ 4.1796875 disk.sda.in‚3.546875 disk.rbd0.in‚0 disk.md0p1.in‚ 4.1796875 disk.sdb.in‚ 0.6328125 disk.sda2.out‚ 841.58203125 network.sent‚ 1.6429065e+07 disk.md0.out‚ 809.59765625 disk.rbd2.in‚0) proc.memory.availableMB‚ 1398.97265625) disk./.usagePercent‚75.80448418369366 disk.sdb2.in‚ 0.6328125 disk.sdb2.out‚ 841.58203125) proc.memory.percent‚18.06325912475586 cpu‚25.182919926339014™4ò†2’- _wandbruntime‚7564  _runtime‚7564FN‘Ç“Á % _timestamp‚1.7885035065816214e+09 _runtime‚7564.358935175 _step‚527 & lr-AdamW/pg1‚0.0005566196681351586 & lr-AdamW/pg2‚0.0005566196681351586 trainer/global_step‚6449èÕSÌ”Æ & lr-AdamW/pg1‚0.0005566196681351586 & lr-AdamW/pg2‚0.0005566196681351586 trainer/global_step‚6449 % _timestamp‚1.7885035065816214e+09 _runtime‚7564.358935175 _step‚527bõ˜œ2•- _wandbruntime‚7564  _runtime‚7564ä ü'–öO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚550949354955264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚550949354955264 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259013632 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 134917732 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚543559128071680 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚543559146074624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235073536 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 _step‚528 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120010608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7720795347456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7389298909184 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚550948426980864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚599701087512064 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39172278 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚630 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80838330 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚525569559596944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7390208880640 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120009978 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7389298909184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235073536 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 _step‚528 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120010608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7720795347456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7389298909184 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚550948426980864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚599701087512064 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39172278 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚630 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80838330 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚525569559596944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7390208880640 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120009978 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7389298909184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1849538560 ´ð2˜- _wandbruntime‚7565  _runtime‚7565â“ÁT'™€N WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚543640747157504 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7391330354176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 298330665 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134937894 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120027810 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚599791289276416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚543640747157504 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7391330354176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39178029 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 % _timestamp‚1.7885035075820823e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163392773 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7390420369408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚607513268826624 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 298330665 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚607513016117760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚525648577191488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚612 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39177417 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚6449 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚551032095514624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 298357037 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚543640765160448 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80850393 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7721955610112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚551032095514624 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 298357037 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80850395 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134937892 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163419143 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784{åƒ 'š…N D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80850395 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134937892 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163419143 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚543640747157504 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7391330354176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 298330665 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134937894 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120027810 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚599791289276416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚543640747157504 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7391330354176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39178029 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 % _timestamp‚1.7885035075820823e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163392773 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7390420369408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚607513268826624 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 298330665 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚607513016117760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚525648577191488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚612 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39177417 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚6449 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚551032095514624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 298357037 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚543640765160448 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80850393 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7721955610112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚551032095514624 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 298357037 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F network.recv‚ 2.515555e+06 disk.md0p1.in‚ 4.1796875 disk.sdb.in‚ 0.6328125 disk.nvme0n1.in‚ 939.5390625 disk.sdb1.in‚0 disk.sdb.out‚844.47998046875! disk.sda2.out‚844.47998046875 disk.sda.out‚844.47998046875 network.sent‚ 1.6539753e+07) disk./.usagePercent‚75.78824010463313ìµ£K¡‚(:„ ìËéÔ’šÎm gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.56' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.14 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R0“¢‚(:ˆ ûËéÔÚ®Âm gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚177.071' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.26775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JóÝV£‚(:Ë ûËéÔÛˆ¡n disk.sdb1.in‚0 network.recv‚ 2.519683e+06 disk.sdb.out‚846.4912109375 disk.sda2.in‚3.546875 disk.sda.out‚846.4912109375 disk.sda1.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 939.5390625# disk.nvme1n1.out‚35436.19140625 disk.rbd2.out‚2.1875 disk.sdb2.out‚846.4912109375! proc.memory.rssMB‚ 2959.484375 cpu‚25.181869249099783 proc.cpu.threads‚17 disk.sda2.out‚846.4912109375 disk.md0p1.out‚814.3125 disk.dm-0.in‚ 1683.86328125# memory_percent‚91.4632797241211) proc.memory.percent‚18.06325912475586 disk.nvme1n1.in‚ 744.32421875 disk.md0.in‚ 4.1796875 disk.md0.out‚814.3125 disk.sdb2.in‚ 0.6328125! disk.dm-0.out‚113134.51953125& proc.memory.availableMB‚ 1398.65625 disk.rbd0.out‚2.1875 disk.md0p1.in‚ 4.1796875 network.sent‚ 1.6547936e+07) disk./.usagePercent‚75.79043869818175! disk.nvme0n1.out‚ 77698.328125 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.sdb.in‚ 0.6328125 disk.sda1.out‚0$ disk./.usageGB‚2710.633197784424 disk.sda.in‚3.546875!Á ´2¤- _wandbruntime‚7619  _runtime‚7619Âf<€Æ¥À trainer/global_step‚6499 % _timestamp‚1.7885035621560361e+09 _runtime‚ 7619.93327267 _step‚531 & lr-AdamW/pg1‚0.0005490335207681783 & lr-AdamW/pg2‚0.0005490335207681783ƒ0p‚˦Š& lr-AdamW/pg2‚0.0005490335207681783 trainer/global_step‚6499 % _timestamp‚1.7885035621560361e+09 _runtime‚ 7619.93327267 _step‚531 & lr-AdamW/pg1‚0.0005490335207681783“U¶þ2§- _wandbruntime‚7619  _runtime‚7619êž{Iü'¨öO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚555085468697088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39459656 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 164639188 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚604212372734464 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7418455993188 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚81441580 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚547640100365824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚555085468697088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120901867 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39460287 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7779118953472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7445386334208 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚547640082362880 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚536938895317332 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7417548018184 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7446296305664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚633 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81441578 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚536937970302968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚547640082362880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7778890171392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120901234 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 300538576 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7445386334208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135925734ZA€(©ûO OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7418455993188 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚81441580 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚547640100365824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚555085468697088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120901867 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39460287 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7779118953472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7445386334208 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚547640082362880 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚536938895317332 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7417548018184 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7446296305664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚633 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81441578 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚536937970302968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚547640082362880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7778890171392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120901234 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 300538576 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7445386334208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135925734 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚555085468697088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39459656 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 164639188 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚604212372734464 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570”¤vü“ª‚(:ˆ ŠÌéÔ”’‡m gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚125.271' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.31775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 §µ-g«‚(:Ü ŠÌéÔƒÌn disk.sda1.in‚0! disk.dm-0.out‚113451.84765625 disk.md0.out‚ 815.89453125 network.recv‚ 2.524218e+06 disk.rbd0.in‚0 disk.sdb1.out‚0! proc.memory.rssMB‚ 2959.484375 network.sent‚ 1.6564896e+07 disk.sdb.in‚ 0.6328125 disk.sda.in‚3.546875$ memory_percent‚91.46473407745361) proc.memory.percent‚18.06325912475586) disk./.usagePercent‚75.79137997737773 disk.md0p1.in‚ 4.1796875! disk.sda2.out‚848.13720703125 disk.md0.in‚ 4.1796875 disk.rbd2.in‚0 disk.sda.out‚848.13720703125 cpu‚25.181542841487033 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sdb.out‚848.13720703125" disk.nvme0n1.out‚ 78015.3828125! disk.sdb2.out‚848.13720703125) proc.memory.availableMB‚ 1398.41796875# disk.nvme1n1.out‚35436.46484375$ disk./.usageGB‚2710.666862487793 disk.rbd0.out‚2.1875 disk.md0p1.out‚ 815.89453125 disk.dm-0.in‚ 1683.86328125 disk.sda2.in‚3.546875 disk.nvme0n1.in‚ 939.5390625 disk.nvme1n1.in‚ 744.32421875 disk.sdb2.in‚ 0.6328125 disk.sdb1.in‚0 disk.rbd2.out‚2.1875O(½Ä2¬- _wandbruntime‚7620  _runtime‚7620û KÊ­€N H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚616 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚614 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135945892 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39465314 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164664045 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚547721701448704 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 164664045 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120919573 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120918957 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚612082627806208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚604302574498816 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚529599439879328 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 % _timestamp‚1.7885035631866724e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135945894 trainer/global_step‚6499 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300609939 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚555169137230848 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚529599456918688 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚547721719451648 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚555168209243136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7446507794432 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7780053307392 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164637675 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚537018105170790 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164637675 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚537018105170790 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135945892 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39465314 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164664045 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚547721701448704 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 164664045 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120919573 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120918957 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚612082627806208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚604302574498816 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚529599439879328 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 % _timestamp‚1.7885035631866724e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135945894 trainer/global_step‚6499 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300609939 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚555169137230848 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚529599456918688 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚547721719451648 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚555168209243136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7446507794432 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7780053307392 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521•;_$Ž2¯- _wandbruntime‚7620  _runtime‚7620¼ŠiZ°Ô 0loss/FRt1/assignment_loss‚2.3028504498626874 0loss/FRt4/assignment_loss‚3.3178977412836894 1loss/SRqqt3/detection_loss‚0.8659045628607273 /loss/FBt1/detection_loss‚0.6277233364836623 /loss/FBt4/detection_loss‚1.9420278314501047 % _timestamp‚1.7885035631895905e+09 0loss/FBt1/assignment_loss‚0.1838113348134442 _step‚534 /loss/FBt2/detection_loss‚1.9413078874349594 trainer/global_step‚6499 0loss/FBt4/assignment_loss‚0.2238825709869161 0loss/FRt2/assignment_loss‚4.0167762894256445 /loss/FRt3/assignment_loss‚1.744545746766604 2loss/SRqqt1/assignment_loss‚1.5409141721633763 1loss/SRqqt2/detection_loss‚2.6015083573081275 %loss/total_loss‚42.03642422502179 2loss/SRqqt4/assignment_loss‚2.2341197030618787 .loss/FRt4/detection_loss‚3.076203480362892 /loss/FBt3/detection_loss‚1.7408575075013297 /loss/FRt3/detection_loss‚1.5631927956182223 epoch‚5 _runtime‚7620.966737918 0loss/SRqqt1/detection_loss‚2.434655093229734 0loss/FBt3/assignment_loss‚0.2716475670440559 .loss/FRt2/detection_loss‚2.133096444548345 2loss/SRqqt2/assignment_loss‚1.8746876191686501 2loss/SRqqt3/assignment_loss‚0.9389358616750687 1loss/FBt2/assignment_loss‚0.35773902354048914 /loss/FRt1/detection_loss‚1.6866914475483454 /loss/SRqqt4/detection_loss‚2.41544740088284MïªÀ_±Ù epoch‚5 _runtime‚7620.966737918 0loss/SRqqt1/detection_loss‚2.434655093229734 0loss/FBt3/assignment_loss‚0.2716475670440559 .loss/FRt2/detection_loss‚2.133096444548345 2loss/SRqqt2/assignment_loss‚1.8746876191686501 2loss/SRqqt3/assignment_loss‚0.9389358616750687 1loss/FBt2/assignment_loss‚0.35773902354048914 /loss/FRt1/detection_loss‚1.6866914475483454 /loss/SRqqt4/detection_loss‚2.41544740088284 0loss/FRt1/assignment_loss‚2.3028504498626874 0loss/FRt4/assignment_loss‚3.3178977412836894 1loss/SRqqt3/detection_loss‚0.8659045628607273 /loss/FBt1/detection_loss‚0.6277233364836623 /loss/FBt4/detection_loss‚1.9420278314501047 % _timestamp‚1.7885035631895905e+09 0loss/FBt1/assignment_loss‚0.1838113348134442 _step‚534 /loss/FBt2/detection_loss‚1.9413078874349594 trainer/global_step‚6499 0loss/FBt4/assignment_loss‚0.2238825709869161 0loss/FRt2/assignment_loss‚4.0167762894256445 /loss/FRt3/assignment_loss‚1.744545746766604 2loss/SRqqt1/assignment_loss‚1.5409141721633763 1loss/SRqqt2/detection_loss‚2.6015083573081275 %loss/total_loss‚42.03642422502179 2loss/SRqqt4/assignment_loss‚2.2341197030618787 .loss/FRt4/detection_loss‚3.076203480362892 /loss/FBt3/detection_loss‚1.7408575075013297 /loss/FRt3/detection_loss‚1.5631927956182223–> i“²‚(:ˆ ™ÌéÔ™Òµl gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.727' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.18175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G‡íb³‚(:× ™ÌéÔá‹Õl disk.md0p1.out‚ 817.234375# disk.nvme0n1.out‚78103.29296875 disk.md0.out‚ 817.234375 disk.sdb1.out‚0 disk.sdb.out‚849.54931640625$ memory_percent‚91.46502017974854) proc.memory.percent‚18.06325912475586 network.recv‚ 2.528797e+06 disk.rbd0.in‚0) proc.memory.availableMB‚ 1398.37109375) disk./.usagePercent‚75.79754932921618 disk.md0.in‚ 4.1796875 disk.sda.in‚3.546875 network.sent‚ 1.6647704e+07# disk.nvme1n1.out‚35553.05859375! disk.sdb2.out‚849.54931640625 cpu‚25.18120061732827 disk.sda2.in‚3.546875 disk.rbd2.in‚0 disk.sda.out‚849.54931640625 disk.sda1.out‚0 disk.sdb1.in‚0! disk.sda2.out‚849.54931640625 disk.nvme0n1.in‚ 939.5390625 disk.nvme1n1.in‚ 744.32421875 disk.sdb2.in‚ 0.6328125 disk.md0p1.in‚ 4.1796875 disk.dm-0.in‚ 1683.86328125 disk.sda1.in‚0 disk.rbd0.out‚2.1875 disk.sdb.in‚ 0.6328125! proc.memory.rssMB‚ 2959.484375 proc.cpu.threads‚17$ disk./.usageGB‚2710.887508392334 disk.rbd2.out‚2.1875 disk.dm-0.out‚113656.3515625®ÝË/Y´‚(:Î ¨ÌéÔþÊöl! proc.memory.rssMB‚ 2959.484375$ disk./.usageGB‚2710.887481689453 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb.out‚851.3408203125 disk.sda2.in‚3.546875 disk.sda.in‚3.546875 disk.sdb1.in‚0 disk.sdb.in‚ 0.6328125 disk.sda2.out‚851.3408203125 disk.nvme0n1.in‚ 939.5390625 disk.md0.out‚ 818.953125 proc.cpu.threads‚17 disk.sdb2.out‚851.3408203125 disk.rbd0.out‚2.1875 disk.dm-0.out‚114031.1171875 cpu‚25.18087727034306 network.recv‚ 2.534083e+06) disk./.usagePercent‚75.79754858259246# disk.nvme0n1.out‚78413.66796875 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚ 4.1796875 disk.md0p1.out‚ 818.953125 disk.rbd2.in‚0 disk.rbd2.out‚2.1875$ memory_percent‚91.46499633789062) proc.memory.percent‚18.06325912475586 disk.sdb2.in‚ 0.6328125 disk.nvme1n1.in‚ 744.32421875# disk.nvme1n1.out‚35617.44921875$ proc.memory.availableMB‚1398.375 network.sent‚ 1.6747011e+07 disk.sda.out‚851.3408203125 disk.dm-0.in‚ 1683.86328125 disk.md0.in‚ 4.1796875÷B¦O’µ‚(:‡ ¨ÌéÔñ–…m gpu.0.gpu‚3.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚117.525' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.38125 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?<‘¶‚(:† ·ÌéÔæݱl gpu.0.gpu‚11.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚184.816' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚46.204 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†«Oge·‚(:Ú ·ÌéÔ’–üo disk.rbd0.out‚2.1875! disk.dm-0.out‚114129.71484375! disk.sda2.out‚853.34521484375# disk.nvme0n1.out‚78512.11328125 disk.rbd2.out‚2.1875 disk.sdb1.out‚0) proc.memory.percent‚18.06325912475586 proc.cpu.threads‚17 disk.sda.in‚3.546875 disk.sda1.in‚0% disk./.usageGB‚2711.1385192871094 disk.nvme1n1.in‚ 744.32421875 disk.md0.out‚ 820.88671875 disk.sdb2.in‚ 0.6328125 disk.sda.out‚853.34521484375) disk./.usagePercent‚75.80456769889032 disk.md0p1.in‚ 4.1796875 disk.nvme0n1.in‚ 939.5390625 disk.rbd2.in‚0 disk.sdb1.in‚0 network.sent‚ 1.6752838e+07 disk.dm-0.in‚ 1683.86328125 disk.sdb.in‚ 0.6328125! disk.sdb2.out‚853.34521484375 disk.sda1.out‚0 disk.rbd0.in‚0 disk.md0.in‚ 4.1796875( proc.memory.availableMB‚ 1398.6171875! proc.memory.rssMB‚ 2959.484375 cpu‚25.18050379813601 network.recv‚ 2.538228e+06 disk.md0p1.out‚ 820.88671875 disk.sdb.out‚853.34521484375 disk.sda2.in‚3.546875" disk.nvme1n1.out‚ 35617.6015625# memory_percent‚91.4635181427002= å‘2¸- _wandbruntime‚7676  _runtime‚7676Ñ©BVƹÀ _runtime‚7676.164908357 _step‚535 & lr-AdamW/pg1‚0.0005414426101089547 & lr-AdamW/pg2‚0.0005414426101089547 trainer/global_step‚6549 $ _timestamp‚1.788503618387601e+093¾OÖ˺Š_runtime‚7676.164908357 _step‚535 & lr-AdamW/pg1‚0.0005414426101089547 & lr-AdamW/pg2‚0.0005414426101089547 trainer/global_step‚6549 $ _timestamp‚1.788503618387601e+09—I~|Û2»- _wandbruntime‚7676  _runtime‚7676ŽÍÜû'¼õO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚616561074076672 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7473425273292 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 165857746 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚559222510413312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚608723634016768 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39749535 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚608723657956864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7885036183884041e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚559222510413312 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚551721054657024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136933734 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚559223438387712 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121793707 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚616561330993152 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚551721036654080 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚658 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚533471302011984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82044830 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚559223438387712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7502383730688 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7502383730688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 302817824 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 trainer/global_step‚6549 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136933732 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚559222510413312 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚551721054657024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136933734 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚559223438387712 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121793707 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚616561330993152 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚551721036654080 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚658 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚533471302011984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82044830 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚559223438387712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7502383730688 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7502383730688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 302817824 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 trainer/global_step‚6549 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136933732 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JøI'¿ÿM K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 165882577 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121811607 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302862841 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7838596642304 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚559305250959360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7838829605376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885036194094028e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136953894 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚559306178947072 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚616652713266688 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 165882577 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 136953894 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82056895 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚608813883661312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 165908947 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7502595219456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7474542546570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚551802673742848 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚541025787167670 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚551802655739904 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39755331 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚608813859721216 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7475450521782 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 302836469 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚533550319606528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7503505204224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚551802655739904 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7503505204224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚533550336645888Š´Q 'À„N B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7474542546570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚551802673742848 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚541025787167670 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚551802655739904 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39755331 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚608813859721216 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7475450521782 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 302836469 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚533550319606528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7503505204224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚551802655739904 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7503505204224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚533550336645888 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 165882577 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121811607 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302862841 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7838596642304 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚559305250959360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7838829605376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885036194094028e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136953894 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚559306178947072 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚616652713266688 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 165882577 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 136953894 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82056895 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚608813883661312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 165908947 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7502595219456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712™YçÐ2Á- _wandbruntime‚7677  _runtime‚7677¹Ž´+ZÂÔ _runtime‚7677.189796688 /loss/FBt4/detection_loss‚2.4659989029169083 /loss/FRt1/assignment_loss‚2.439346520602703 0loss/FBt2/assignment_loss‚0.2750747351198015 /loss/FRt3/detection_loss‚1.5559235591913614 trainer/global_step‚6549 0loss/FRt3/assignment_loss‚1.5681076041409667 /loss/FRt2/detection_loss‚2.2312680734321475 1loss/SRqqt2/detection_loss‚2.5459547424316407 /loss/FBt1/detection_loss‚0.9383071528394676 0loss/FBt4/assignment_loss‚0.5651637221219857 _step‚538 1loss/SRqqt1/assignment_loss‚1.421759950518608 2loss/SRqqt3/assignment_loss‚0.9554013368854327 0loss/FRt2/assignment_loss‚3.3231178149580956 0loss/FRt4/assignment_loss‚3.7516841292381287 2loss/SRqqt4/assignment_loss‚1.8222828158842665 1loss/FBt3/assignment_loss‚0.07826053683212111 0loss/SRqqt4/detection_loss‚2.585807440536363 1loss/SRqqt2/assignment_loss‚1.890233235359192 0loss/SRqqt1/detection_loss‚2.451868779659271 1loss/SRqqt3/detection_loss‚0.8162841851920334 .loss/FBt2/detection_loss‚2.306300302967429 /loss/FRt1/detection_loss‚1.9636882603960657 /loss/FBt3/detection_loss‚1.6541887779127469 1loss/FBt1/assignment_loss‚0.09458620769561504 /loss/FRt4/detection_loss‚2.9494804441928864 %loss/total_loss‚42.65008923102524 epoch‚5 $ _timestamp‚1.788503619412169e+09Uè3ì_ÃÙ 0loss/FBt2/assignment_loss‚0.2750747351198015 /loss/FRt3/detection_loss‚1.5559235591913614 trainer/global_step‚6549 0loss/FRt3/assignment_loss‚1.5681076041409667 /loss/FRt2/detection_loss‚2.2312680734321475 1loss/SRqqt2/detection_loss‚2.5459547424316407 /loss/FBt1/detection_loss‚0.9383071528394676 0loss/FBt4/assignment_loss‚0.5651637221219857 _step‚538 1loss/SRqqt1/assignment_loss‚1.421759950518608 2loss/SRqqt3/assignment_loss‚0.9554013368854327 0loss/FRt2/assignment_loss‚3.3231178149580956 0loss/FRt4/assignment_loss‚3.7516841292381287 2loss/SRqqt4/assignment_loss‚1.8222828158842665 1loss/FBt3/assignment_loss‚0.07826053683212111 0loss/SRqqt4/detection_loss‚2.585807440536363 1loss/SRqqt2/assignment_loss‚1.890233235359192 0loss/SRqqt1/detection_loss‚2.451868779659271 1loss/SRqqt3/detection_loss‚0.8162841851920334 .loss/FBt2/detection_loss‚2.306300302967429 /loss/FRt1/detection_loss‚1.9636882603960657 /loss/FBt3/detection_loss‚1.6541887779127469 1loss/FBt1/assignment_loss‚0.09458620769561504 /loss/FRt4/detection_loss‚2.9494804441928864 %loss/total_loss‚42.65008923102524 epoch‚5 $ _timestamp‚1.788503619412169e+09 _runtime‚7677.189796688 /loss/FBt4/detection_loss‚2.4659989029169083 /loss/FRt1/assignment_loss‚2.439346520602703š.Æž’Ä‚(:‡ ÆÌéÔúïèm gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚172.118' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.0295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ#QÉ^Å‚(:Ó ÆÌéÔÜ–‚n network.recv‚ 2.543049e+06$ disk./.usageGB‚2711.138385772705 disk.nvme1n1.in‚ 744.32421875 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sdb1.in‚0 cpu‚25.180165611737895 network.sent‚ 1.6756913e+07 disk.nvme0n1.in‚ 939.5390625 disk.nvme1n1.out‚ 35618.0625 disk.sdb.in‚ 0.6328125 disk.sda2.in‚3.546875 disk.rbd0.out‚2.1875 disk.md0p1.in‚ 4.1796875 disk.md0p1.out‚ 822.32421875 disk.sda1.out‚0! proc.memory.rssMB‚ 2959.484375) disk./.usagePercent‚75.80456396577169! disk.nvme0n1.out‚ 78975.265625! disk.sdb2.out‚854.85302734375 disk.sda.in‚3.546875' proc.memory.availableMB‚ 1398.609375! disk.sda2.out‚854.85302734375 disk.md0.out‚ 822.32421875 disk.rbd2.out‚2.1875 disk.dm-0.out‚ 114593.328125$ memory_percent‚91.46356582641602 disk.md0.in‚ 4.1796875 disk.sdb2.in‚ 0.6328125) proc.memory.percent‚18.06325912475586 disk.sdb.out‚854.85302734375 disk.sda.out‚854.85302734375 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 1683.86328125 proc.cpu.threads‚17²ù–^‘Æ‚(:† ÕÌéÔ¦Á³m gpu.0.gpu‚44.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚177.688' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.422 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yRùl—Ç‚(:Õ ÕÌéÔ¸Ççm disk.sdb1.out‚0 disk.sdb.out‚856.14404296875) proc.memory.percent‚18.06325912475586% disk./.usageGB‚2710.558ŃRMÉ9141845703 disk.sda1.in‚0 disk.nvme0n1.in‚ 939.5390625 disk.nvme1n1.in‚ 744.32421875 proc.cpu.threads‚17 disk.sda.in‚3.546875 disk.sda1.out‚0 disk.sdb1.in‚0 disk.md0.out‚823.5625 disk.rbd2.out‚2.1875 network.sent‚ 1.6866014e+07 disk.sda.out‚856.14404296875 disk.sdb.in‚ 0.6328125 network.recv‚ 2.547535e+06 disk.md0p1.out‚823.5625 disk.md0.in‚ 4.1796875! proc.memory.rssMB‚ 2959.484375 disk.md0p1.in‚ 4.1796875 disk.dm-0.in‚ 1683.86328125 disk.sda2.in‚3.546875) proc.memory.availableMB‚ 1398.83984375 cpu‚25.179816910138214! disk.sdb2.out‚856.14404296875 disk.rbd0.out‚2.1875! disk.dm-0.out‚114947.75390625! disk.sda2.out‚856.14404296875" disk.nvme0n1.out‚ 79329.4921875 disk.rbd2.in‚0$ memory_percent‚91.46215915679932 disk.sdb2.in‚ 0.6328125# disk.nvme1n1.out‚35618.26171875) disk./.usagePercent‚75.78836169763974 disk.rbd0.in‚0y~ò¶“È‚(:ˆ äÌéÔß¼Ãk gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚172.727' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.18175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜6€W^É‚(:Ó äÌéÔûôâl network.recv‚ 2.551995e+06 disk.sda.in‚3.546875 network.sent‚ 1.6872103e+07) disk./.usagePercent‚75.78970732691299 disk.sdb2.in‚ 0.6328125 disk.sdb2.out‚858.2060546875 disk.rbd0.out‚2.1875 disk.sdb1.in‚0 disk.nvme1n1.in‚ 744.32421875 disk.rbd2.out‚2.1875 disk.nvme0n1.in‚ 939.5390625 disk.md0.in‚ 4.1796875 disk.md0.out‚ 825.5703125! proc.memory.rssMB‚ 2959.484375 disk.rbd0.in‚0 disk.md0p1.in‚ 4.1796875 disk.md0p1.out‚ 825.5703125 disk.dm-0.in‚ 1683.86328125 disk.sdb.out‚858.2060546875 disk.sda2.in‚3.546875 disk.sda2.out‚858.2060546875% disk./.usageGB‚2710.6070404052734 disk.sda.out‚858.2060546875 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme1n1.out‚ 35618.4921875 disk.rbd2.in‚0) proc.memory.availableMB‚ 1398.30859375 disk.sdb1.out‚0 disk.sdb.in‚ 0.6328125) proc.memory.percent‚18.06325912475586 proc.cpu.threads‚17 disk.dm-0.out‚ 114997.453125" disk.nvme0n1.out‚ 79378.9609375# memory_percent‚91.4654016494751 cpu‚25.179475297733873VÀª’Ê‚(:‡ óÌéÔ¯ûk gpu.0.gpu‚57.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚185.426' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚46.3565 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉDKi^Ë‚(:Ó óÌéÔ¿‹ðl network.recv‚ 2.556036e+06 disk.sda.in‚3.546875 disk.md0p1.in‚ 4.1796875 disk.md0p1.out‚ 827.09765625) proc.memory.availableMB‚ 1398.30859375 disk.sda.out‚859.7978515625 disk.dm-0.in‚ 1683.86328125 disk.sda2.out‚859.7978515625 disk.md0.out‚ 827.09765625 disk.sdb2.in‚ 0.6328125 disk.sdb2.out‚859.7978515625 disk.sdb.out‚859.7978515625 disk.nvme0n1.in‚ 939.5390625 disk.md0.in‚ 4.1796875 disk.rbd0.in‚0 disk.sda2.in‚3.546875# memory_percent‚91.4654016494751 network.sent‚ 1.6875746e+07# disk.nvme0n1.out‚79589.25390625 disk.sdb1.in‚0! proc.memory.rssMB‚ 2959.484375) proc.memory.percent‚18.06325912475586% disk./.usageGB‚2710.5564460754395 disk.rbd2.in‚0 disk.sdb1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚75.78829268827546 disk.nvme1n1.out‚35618.75 disk.sdb.in‚ 0.6328125 cpu‚25.179127246384432! disk.dm-0.out‚115208.00390625 disk.nvme1n1.in‚ 744.32421875 disk.rbd2.out‚2.1875 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚2.1875;Z”£2Ì- _wandbruntime‚7731  _runtime‚7731Ç?˜ÇÍÁ & lr-AdamW/pg1‚0.0005338485180918916 & lr-AdamW/pg2‚0.0005338485180918916 trainer/global_step‚6599 % _timestamp‚1.7885036736036766e+09 _runtime‚7731.380968631 _step‚539:ý„ÌÎÆ & lr-AdamW/pg1‚0.0005338485180918916 & lr-AdamW/pg2‚0.0005338485180918916 trainer/global_step‚6599 % _timestamp‚1.7885036736036766e+09 _runtime‚7731.380968631 _step‚539›O- é2Ï- _wandbruntime‚7731  _runtime‚7731'Våkü'ÐöO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚563360480103936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167128992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚621130882899456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚613234943179264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137941734 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚555801990945280 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 167128992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 122686374 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚537422198778544 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7895939720192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214102016 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 122686979 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚544951484267584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 _runtime‚7731.382100548 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚537422181739184 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 % _timestamp‚1.7885036736047966e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 167102648 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚563359552129536 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚555802008948224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82648080 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚605 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚544952409281948 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7529302528400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚563359552129536 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 167102648 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚555801990945280 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7558471155712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7558471155712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _step‚540 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7895725618176 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40038296 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚621130644857344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82648078 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚603 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚563360480103936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7557561184256 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚555802008948224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137941732 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40038899 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚563359552129536 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚555802008948224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82648080 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚605 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚544952409281948 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7529302528400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚563359552129536 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 167102648 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚555801990945280 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7558471155712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7558471155712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _step‚540 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7895725618176 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40038296 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚621130644857344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82648078 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚603 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚563360480103936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7557561184256 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚555802008948224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137941732 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40038899 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚537422198778544 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7895939720192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214102016 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 122686979 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚544951484267584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 _runtime‚7731.382100548 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚537422181739184 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 % _timestamp‚1.7885036736047966e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 167102648œÀ^92Ò- _wandbruntime‚7732  _runtime‚7732À!ÕÓ€N F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚555883610031104 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122704154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885036746241267e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40044011 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚545032544149978 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚537501216373088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305115743 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7558682644480 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7896905165824 trainer/global_step‚6599 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 305089371 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82660145 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7558682644480 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 122704773 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚613325144943616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 305115743 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40044628 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚555883610031104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570"ü‰! 'Ô…N ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚555883610031104 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122704154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885036746241267e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40044011 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚545032544149978 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚537501216373088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305115743 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7558682644480 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7896905165824 trainer/global_step‚6599 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 305089371 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82660145 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7558682644480 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 122704773 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚613325144943616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 305115743 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40044628 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚555883610031104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144;Ø4#2Õ- _wandbruntime‚7732  _runtime‚7732 °,XÖÒ _runtime‚7732.404492914 1loss/SRqqt2/assignment_loss‚1.919016359994809 1loss/FBt3/assignment_loss‚0.05791505069937557 /loss/FRt1/detection_loss‚1.9512008833117342 1loss/SRqqt4/detection_loss‚2.4900174019011585 .loss/FBt3/detection_loss‚3.041886794567108 epoch‚5 % _timestamp‚1.7885036746273525e+09 0loss/FRt4/assignment_loss‚3.5807768354813256 1loss/FBt1/assignment_loss‚0.05573604930896566 /loss/FRt4/detection_loss‚2.8316967437664666 0loss/FBt2/assignment_loss‚0.0800625333972943 .loss/FBt2/detection_loss‚1.890378443633809 _step‚542 /loss/FRt2/assignment_loss‚4.148881664081496 0loss/FRt3/assignment_loss‚1.9381102191077337 1loss/SRqqt1/assignment_loss‚1.460529237985611 1loss/SRqqt3/detection_loss‚0.8673866932781843 /loss/FBt1/detection_loss‚0.7708970260040803 0loss/FRt1/assignment_loss‚2.4754965691953092 2loss/SRqqt3/assignment_loss‚1.1044214086033977 /loss/FRt3/detection_loss‚1.6857181621922388 /loss/SRqqt1/detection_loss‚2.36471976518631 .loss/FBt4/detection_loss‚2.335347305644642 1loss/SRqqt2/detection_loss‚2.6074585020542145 %loss/total_loss‚43.61020217496248 1loss/FBt4/assignment_loss‚0.14346136694237802 .loss/FRt2/detection_loss‚2.272165512856172 trainer/global_step‚6599 1loss/SRqqt4/assignment_loss‚1.536921645768664Zò`]×× 0loss/FBt2/assignment_loss‚0.0800625333972943 .loss/FBt2/detection_loss‚1.890378443633809 _step‚542 /loss/FRt2/assignment_loss‚4.148881664081496 0loss/FRt3/assignment_loss‚1.9381102191077337 1loss/SRqqt1/assignment_loss‚1.460529237985611 1loss/SRqqt3/detection_loss‚0.8673866932781843 /loss/FBt1/detection_loss‚0.7708970260040803 0loss/FRt1/assignment_loss‚2.4754965691953092 2loss/SRqqt3/assignment_loss‚1.1044214086033977 /loss/FRt3/detection_loss‚1.6857181621922388 /loss/SRqqt1/detection_loss‚2.36471976518631 .loss/FBt4/detection_loss‚2.335347305644642 1loss/SRqqt2/detection_loss‚2.6074585020542145 %loss/total_loss‚43.61020217496248 1loss/FBt4/assignment_loss‚0.14346136694237802 .loss/FRt2/detection_loss‚2.272165512856172 trainer/global_step‚6599 1loss/SRqqt4/assignment_loss‚1.536921645768664 _runtime‚7732.404492914 1loss/SRqqt2/assignment_loss‚1.919016359994809 1loss/FBt3/assignment_loss‚0.05791505069937557 /loss/FRt1/detection_loss‚1.9512008833117342 1loss/SRqqt4/detection_loss‚2.4900174019011585 .loss/FBt3/detection_loss‚3.041886794567108 epoch‚5 % _timestamp‚1.7885036746273525e+09 0loss/FRt4/assignment_loss‚3.5807768354813256 1loss/FBt1/assignment_loss‚0.05573604930896566 /loss/FRt4/detection_loss‚2.8316967437664666ž•ðeé“Ø‚(:ˆ ‚ÍéÔ¦í©l gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚124.661' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.16525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[ÛÑÂ`Ù‚(:Õ ‚ÍéÔŽÕÄl) proc.memory.percent‚18.06325912475586$ memory_percent‚91.46380424499512 network.recv‚ 2.560751e+06 disk.md0p1.in‚ 4.1796875 disk.sdb.in‚ 0.6328125 disk.sdb.out‚861.2568359375 disk.nvme0n1.in‚ 939.5390625 disk.rbd2.out‚2.1875 disk.sda.out‚861.2568359375 disk.sdb2.in‚ 0.6328125 disk.sdb2.out‚861.2568359375 disk.sdb1.in‚0 disk.md0.in‚ 4.1796875! proc.memory.rssMB‚ 2959.484375 disk.rbd0.in‚0 disk.md0p1.out‚ 828.4921875 disk.dm-0.in‚ 1683.86328125 disk.dm-0.out‚115446.3046875 disk.sda2.in‚3.546875# disk.nvme1n1.out‚35619.09765625 disk.sda1.in‚0( proc.memory.availableMB‚ 1398.5703125 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.sda.in‚3.546875 network.sent‚ 1.688201e+07% disk./.usageGB‚2710.6374588012695 disk.rbd0.out‚2.1875 disk.sda2.out‚861.2568359375 disk.nvme1n1.in‚ 744.32421875 disk.md0.out‚ 828.4921875 cpu‚25.178783541178113) disk./.usagePercent‚75.79055783799613# disk.nvme0n1.out‚79827.20703125 disk.rbd2.in‚0 disk.sda1.out‚0¿Ó‹L“Ú‚(:ˆ ‘ÍéÔ‘Ÿƒl gpu.0.gpu‚11.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚156.911' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.22775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ´rw_Û‚(:Ô ‘ÍéÔäÿŠm network.recv‚ 2.565306e+06% disk./.usageGB‚2710.7000122070312 disk.sda.in‚3.546875 disk.md0.out‚ 830.4765625! proc.memory.rssMB‚ 2959.484375 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 1683.86328125 disk.sdb.in‚ 0.6328125 disk.nvme1n1.in‚ 744.32421875 proc.cpu.threads‚17 disk.sda.out‚863.3037109375 disk.md0p1.in‚ 4.1796875) proc.memory.percent‚18.06325912475586 cpu‚25.178467267663457) disk./.usagePercent‚75.79230685740187 disk.sda1.in‚0 disk.rbd0.out‚2.1875 disk.md0p1.out‚ 830.4765625 disk.sdb.out‚863.3037109375" disk.nvme1n1.out‚ 35619.2421875 disk.rbd2.in‚0$ memory_percent‚91.46559238433838 disk.sdb1.in‚0 disk.sda2.in‚3.546875 disk.sda2.out‚863.3037109375 disk.sdb2.in‚ 0.6328125 network.sent‚ 1.6990973e+07 disk.dm-0.out‚115635.4453125! disk.nvme0n1.out‚ 80016.203125 disk.md0.in‚ 4.1796875 disk.rbd2.out‚2.1875) proc.memory.availableMB‚ 1398.27734375 disk.sdb2.out‚863.3037109375 disk.nvme0n1.in‚ 939.5390625nMN2“Ü‚(:ˆ  ÍéÔüäÔk gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆÉÝ£eÝ‚(:Ú  ÍéÔÆ“Ùl disk.sdb2.in‚ 0.6328125) proc.memory.percent‚18.06325912475586 disk.sda1.out‚0 disk.rbd0.out‚2.1875 network.sent‚ 1.6994979e+07 disk.md0p1.in‚ 4.1796875 disk.md0p1.out‚ 832.1328125! disk.sda2.out‚865.02001953125 disk.rbd2.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 1683.86328125! disk.dm-0.out‚116072.64453125 disk.sdb.in‚ 0.6328125 disk.nvme0n1.in‚ 939.5390625 cpu‚25.178126498915727 proc.cpu.threads‚17 disk.sda2.in‚3.546875! disk.sdb2.out‚865.02001953125 disk.sdb1.out‚0 disk.md0.out‚ 832.1328125 disk.sda.in‚3.546875 disk.rbd0.in‚0$ memory_percent‚91.46707057952881% disk./.usageGB‚2711.0124740600586 disk.nvme1n1.in‚ 744.32421875# disk.nvme1n1.out‚35735.91015625) disk./.usagePercent‚75.80104342158795! disk.nvme0n1.out‚ 80336.734375 disk.md0.in‚ 4.1796875! proc.memory.rssMB‚ 2959.484375 network.recv‚ 2.569986e+06 disk.sda.out‚865.02001953125 disk.sdb1.in‚0) proc.memory.availableMB‚ 1398.03515625 disk.sdb.out‚865.02001953125 disk.rbd2.out‚2.1875¼‘“Þ‚(:ˆ ¯ÍéÔ×äm gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚161.091' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.27275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰´MÍZß‚(:Ï ¯ÍéÔØó–n disk.md0p1.in‚ 4.1796875 disk.rbd2.out‚2.1875) proc.memory.availableMB‚ 1398.53515625 proc.cpu.threads‚17 network.recv‚ 2.573349e+06 disk.rbd0.in‚0# disk.nvme0n1.out‚80529.81640625 disk.sda2.out‚ 866.873046875 disk.nvme0n1.in‚ 939.5390625 disk.md0.in‚ 4.1796875 disk.rbd2.in‚0! proc.memory.rssMB‚ 2959.484375( disk./.usagePercent‚75.8045656723402 disk.sda2.in‚3.546875$ memory_percent‚91.46401882171631 disk.sda1.in‚0 disk.sdb1.out‚0 disk.md0p1.out‚ 833.9296875 disk.sdb.out‚ 866.873046875" disk.nvme1n1.out‚ 35736.0703125% disk./.usageGB‚2711.1384468078613 disk.sda1.out‚0 disk.rbd0.out‚2.1875 disk.nvme1n1.in‚ 744.32421875 disk.sda.out‚ 866.873046875 network.sent‚ 1.699979e+07 disk.sdb.in‚ 0.6328125) proc.memory.percent‚18.06325912475586 disk.dm-0.in‚ 1683.86328125! disk.dm-0.out‚116265.88671875 disk.md0.out‚ 833.9296875 disk.sdb2.in‚ 0.6328125 disk.sdb2.out‚ 866.873046875 disk.sda.in‚3.546875 cpu‚25.1778386657818 disk.sdb1.in‚0¯UÖh2à- _wandbruntime‚7787  _runtime‚7787¦v­rÇáÁ % _timestamp‚1.7885037299247963e+09 _runtime‚7787.701971154 _step‚543 & lr-AdamW/pg1‚0.0005262528273143829 & lr-AdamW/pg2‚0.0005262528273143829 trainer/global_step‚6649 ÅEÌâÆ % _timestamp‚1.7885037299247963e+09 _runtime‚7787.701971154 _step‚543 & lr-AdamW/pg1‚0.0005262528273143829 & lr-AdamW/pg2‚0.0005262528273143829 trainer/global_step‚6649ŸÛ"O"2ã- _wandbruntime‚7787  _runtime‚7787$àoƒü'äöO QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138949732 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83251330 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚548959166264256 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚567496593845760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚625700614503936 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7954410042368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7614558580736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚559882963239424 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 % _timestamp‚1.7885037299255342e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚617746204461568 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7585179783508 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚544 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 307297282 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚567497521820160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚541373061466384 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚625700869323264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚559882945236480 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138949734 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123580565 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138949732 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7614558580736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 168347550 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚635 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 _runtime‚7787.702905218 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚548958241249892 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168347550 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚559882963239424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83251328 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚567496593845760 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0¤ðZ›åûO G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40329235 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40328602 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7954640921600 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 Q®m&f%CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 307323628 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚559882945236480 trainer/global_step‚6649 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7613648609280 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 168373894 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 123579930 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83251330 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚548959166264256 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚567496593845760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚625700614503936 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7954410042368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7614558580736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚559882963239424 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 % _timestamp‚1.7885037299255342e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚617746204461568 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7585179783508 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚544 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 307297282 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚567497521820160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚541373061466384 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚625700869323264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚559882945236480 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138949734 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123580565 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138949732 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7614558580736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 168347550 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚635 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 _runtime‚7787.702905218 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚548958241249892 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168347550 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚559882963239424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83251328 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚567496593845760 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138949732 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307368645 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7614770069504 % _timestamp‚1.7885037310045416e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7615680054272 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123597712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138969894 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40334319 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚559964564322304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚559964582325248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 168398751 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚549039301132286 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚559964582325248 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138969894 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7586297056786 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 168398751 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 168372381 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 168372381 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7955575613440 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚549038376117714 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7614770069504 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚617836454106112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 trainer/global_step‚6649 _runtime‚7788.783413479 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚559964564322304 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚545 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123598360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 307368645 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚541452096100288 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚541452079060928 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚617836430166016 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7955804382208 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 307342273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚567580262379520 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83263395 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83263393 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138969892 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚646 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40334965 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 11953766409Mv‡ 'è…N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40334319 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚559964564322304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚559964582325248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 168398751 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚549039301132286 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚559964582325248 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138969894 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7586297056786 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 168398751 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 168372381 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 168372381 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7955575613440 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚549038376117714 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7614770069504 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚617836454106112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 trainer/global_step‚6649 _runtime‚7788.783413479 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚559964564322304 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚545 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123598360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 307368645 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚541452096100288 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚541452079060928 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚617836430166016 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7955804382208 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 307342273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚567580262379520 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83263395 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83263393 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138969892 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚646 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40334965 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚625792005779456 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138969892 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚567579334391808 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307368645 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7614770069504 % _timestamp‚1.7885037310045416e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7615680054272 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123597712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138969894 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712¡zIª2é- _wandbruntime‚7788  _runtime‚7788R¾u…XêÒ % _timestamp‚1.7885037310077353e+09 0loss/FRt2/assignment_loss‚3.5109410755909405 .loss/FRt2/detection_loss‚2.258236116514756 ,loss/FRt4/detection_loss‚2.9459085115071 2loss/SRqqt2/assignment_loss‚1.1452470590670905 2loss/SRqqt3/assignment_loss‚1.1724849711779668 2loss/FBt3/assignment_loss‚0.047342227333501796 /loss/FRt3/detection_loss‚1.6673226410194355 _runtime‚7788.784811606 0loss/FRt3/assignment_loss‚1.7869587987661362 2loss/SRqqt4/assignment_loss‚2.4685302767902613 1loss/SRqqt2/detection_loss‚2.3127731641133624 0loss/SRqqt4/detection_loss‚2.649620061474187 .loss/FBt4/detection_loss‚2.562730425596237 /loss/FRt1/assignment_loss‚2.457371717844254 1loss/SRqqt1/assignment_loss‚1.602191355234633 1loss/FBt1/assignment_loss‚0.05036602482152252 /loss/FRt1/detection_loss‚1.7907656589829504 %loss/total_loss‚42.00877346031335 trainer/global_step‚6649 0loss/SRqqt1/detection_loss‚2.181437212228775 /loss/FBt1/detection_loss‚0.8816981898893018 epoch‚5 /loss/FBt2/detection_loss‚2.4462457266118793 _step‚546 /loss/FRt4/assignment_loss‚3.139497132136904 0loss/FBt2/assignment_loss‚0.2674309624934621 0loss/FBt4/assignment_loss‚0.0833405686772494 1loss/SRqqt3/detection_loss‚0.8918292553528495 /loss/FBt3/detection_loss‚1.6885043270885944‚šÏZ]ë× 0loss/FRt2/assignment_loss‚3.5109410755909405 .loss/FRt2/detection_loss‚2.258236116514756 ,loss/FRt4/detection_loss‚2.9459085115071 2loss/SRqqt2/assignment_loss‚1.1452470590670905 2loss/SRqqt3/assignment_loss‚1.1724849711779668 2loss/FBt3/assignment_loss‚0.047342227333501796 /loss/FRt3/detection_loss‚1.6673226410194355 _runtime‚7788.784811606 0loss/FRt3/assignment_loss‚1.7869587987661362 2loss/SRqqt4/assignment_loss‚2.4685302767902613 1loss/SRqqt2/detection_loss‚2.3127731641133624 0loss/SRqqt4/detection_loss‚2.649620061474187 .loss/FBt4/detection_loss‚2.562730425596237 /loss/FRt1/assignment_loss‚2.457371717844254 1loss/SRqqt1/assignment_loss‚1.602191355234633 1loss/FBt1/assignment_loss‚0.05036602482152252 /loss/FRt1/detection_loss‚1.7907656589829504 %loss/total_loss‚42.00877346031335 trainer/global_step‚6649 0loss/SRqqt1/detection_loss‚2.181437212228775 /loss/FBt1/detection_loss‚0.8816981898893018 epoch‚5 /loss/FBt2/detection_loss‚2.4462457266118793 _step‚546 /loss/FRt4/assignment_loss‚3.139497132136904 0loss/FBt2/assignment_loss‚0.2674309624934621 0loss/FBt4/assignment_loss‚0.0833405686772494 1loss/SRqqt3/detection_loss‚0.8918292553528495 /loss/FBt3/detection_loss‚1.6885043270885944 % _timestamp‚1.7885037310077353e+09¢)º38ì‚(:‡ ¾ÍéÔŒƒˆn gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.942' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4855 gpu.0.smClock‚1410 gpu.0.rý‰ÆZmemoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a8€GRí‚(:Ç ¾ÍéÔ‡Óo) proc.memory.percent‚18.06325912475586 proc.cpu.threads‚17! disk.sdb2.out‚868.51611328125 disk.sda1.in‚0 disk.rbd0.in‚0" disk.nvme1n1.out‚ 35736.3515625 disk.rbd2.in‚0" proc.memory.availableMB‚1398.5 network.recv‚ 2.578156e+06) disk./.usagePercent‚75.80453719397813 disk.sda1.out‚0 disk.sdb1.in‚0 disk.md0p1.out‚835.5# memory_percent‚91.4642333984375 disk.md0.out‚835.5 network.sent‚ 1.7006163e+07 disk.sdb2.in‚ 0.6328125 disk.sda.out‚868.51611328125# disk.nvme0n1.out‚80804.36328125 cpu‚25.177586391889974 disk.dm-0.in‚ 1683.86328125! disk.sda2.out‚868.51611328125% disk./.usageGB‚2711.1374282836914 disk.md0p1.in‚ 4.1796875 disk.sda2.in‚3.546875 disk.md0.in‚ 4.1796875 disk.rbd0.out‚2.1875 disk.sdb1.out‚0 disk.nvme0n1.in‚ 939.5390625! disk.dm-0.out‚116540.71484375 disk.sdb.in‚ 0.6328125 disk.sdb.out‚868.51611328125 disk.nvme1n1.in‚ 744.32421875 disk.sda.in‚3.546875 disk.rbd2.out‚2.1875! proc.memory.rssMB‚ 2959.484375J¹;m“î‚(:ˆ ÍÍéÔ‘Ãm gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚125.603' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.40075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DŸˆ}gï‚(:Ü ÍÍéÔá÷¢n! disk.sdb2.out‚870.37158203125$ memory_percent‚91.46411418914795" disk.nvme1n1.out‚ 35736.5234375 disk.sdb2.in‚ 0.6328125! proc.memory.rssMB‚ 2959.484375 network.sent‚ 1.7115246e+07( disk./.usagePercent‚75.8045377272808 disk.sdb.out‚870.37158203125 disk.sda2.in‚3.546875 disk.md0.out‚ 837.29296875 disk.dm-0.in‚ 1683.86328125 disk.md0p1.out‚ 837.29296875 disk.sdb.in‚ 0.6328125! disk.sda2.out‚870.37158203125 disk.md0.in‚ 4.1796875 disk.rbd2.out‚2.1875 disk.sda.out‚870.37158203125 disk.nvme0n1.in‚ 939.5390625 disk.rbd2.in‚0 disk.sda.in‚3.546875 disk.sda1.out‚0 disk.md0p1.in‚ 4.1796875% disk./.usageGB‚2711.1374473571777 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0! disk.dm-0.out‚116540.88671875# disk.nvme0n1.out‚80804.36328125 disk.nvme1n1.in‚ 744.32421875 disk.rbd0.out‚2.1875 disk.sdb1.in‚0) proc.memory.availableMB‚ 1398.51953125) proc.memory.percent‚18.06325912475586 cpu‚25.177316734157642 proc.cpu.threads‚17 network.recv‚ 2.582955e+06‘Õ{g‘ð‚(:† ÜÍéÔ‚Ê´m gpu.0.gpu‚4.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚127.446' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.8615 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b1]ñ‚(:Ò ÜÍéÔâþ—n disk.md0p1.in‚ 4.1796875 disk.dm-0.in‚ 1683.86328125" disk.nvme0n1.out‚ 81405.6484375 disk.sda.in‚3.546875) disk./.usagePercent‚75.78971649971876 disk.sdb.out‚871.8974609375" disk.nvme1n1.out‚ 35736.6796875 cpu‚25.177010546155614 network.recv‚ 2.587168e+06 disk.rbd2.out‚2.1875 disk.sda.out‚871.8974609375 disk.sda1.out‚0 disk.sdb1.in‚0 disk.md0p1.out‚ 838.7578125 disk.sda2.out‚871.8974609375 disk.nvme1n1.in‚ 744.32421875 disk.md0.out‚ 838.7578125 disk.rbd2.in‚0 disk.sdb2.in‚ 0.6328125 disk.sdb2.out‚871.8974609375 network.sent‚ 1.7121223e+07 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.dm-0.out‚ 117142.328125) proc.memory.percent‚18.06325912475586 disk.sda2.in‚3.546875 disk.md0.in‚ 4.1796875 disk.sda1.in‚0# memory_percent‚91.4644718170166( proc.memory.availableMB‚ 1398.4609375 disk.rbd0.out‚2.1875! proc.memory.rssMB‚ 2959.484375% disk./.usageGB‚2710.6073684692383 disk.sdb.in‚ 0.6328125 disk.nvme0n1.in‚ 939.5390625 proc.cpu.threads‚17yÐpO2ò- _wandbruntime‚7844  _runtime‚78443Fë¹ÇóÁ trainer/global_step‚6699 % _timestamp‚1.7885037871038806e+09 _runtime‚7844.881161545 _step‚547 & lr-AdamW/pg1‚0.0005186571207070021 & lr-AdamW/pg2‚0.0005186571207070021ò8•hÌôÆ _step‚547 & lr-AdamW/pg1‚0.0005186571207070021 & lr-AdamW/pg2‚0.0005186571207070021 trainer/global_step‚6699 % _timestamp‚1.7885037871038806e+09 _runtime‚7844.881161545£‚U2õ- _wandbruntime‚7844  _runtime‚7844dÜVÆû'öõO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139957734 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 139957732 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚552964998232200 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚571633635561984 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚563963917530624 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 124474021 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚622257513624064 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 trainer/global_step‚6699 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40618799 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 169592452 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚545323958232944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7641965013620 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7670646005760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚571634563536384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚644 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83854578 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚563963899527680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚642 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8013006722048 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169618796 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7669736034304 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40619441 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚622257489683968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7670646005760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚571633635561984 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 124473377 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83854580 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚630270757516800 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 $ _timestamp‚1.788503787104973e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚548 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 139957732 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 169592452 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚571634563536384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚630270496406016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8013243892736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚545323941193584 _runtime‚7844.882474881 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 309576530 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 169618796 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚552965923246564 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 309550184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237170688 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚563963899527680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚563963917530624 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 139957734 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0à˜Å<(÷úO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139957734 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 139957732 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚552964998232200 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚571633635561984 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚563963917530624 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 124474021 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚622257513624064 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 trainer/global_step‚6699 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40618799 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 169592452 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚545323958232944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7641965013620 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7670646005760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚571634563536384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚644 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83854578 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚563963899527680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚642 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8013006722048 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169618796 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7669736034304 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40619441 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚622257489683968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7670646005760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚571633635561984 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 124473377 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83854580 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚630270757516800 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 $ _timestamp‚1.788503787104973e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚548 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 139957732 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 169592452 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚571634563536384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚630270496406016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8013243892736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚545323941193584 _runtime‚7844.882474881 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 309576530 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 169618796 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚552965923246564 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 309550184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237170688 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚563963899527680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚563963917530624 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 139957734 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0¤'D:^ø‚(:Ó ëÍéÔùäân network.sent‚ 1.712489e+07 disk.nvme0n1.in‚ 939.5390625 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.md0p1.in‚ 4.1796875 disk.sdb.in‚ 0.6328125! proc.memory.rssMB‚ 2959.484375 disk.sda1.in‚0 disk.md0p1.out‚ 840.23828125 disk.sdb.out‚ 873.439453125 network.recv‚ 2.591393e+06 disk.sdb2.out‚ 873.439453125$ memory_percent‚91.46440029144287) proc.memory.percent‚18.06325912475586 disk.rbd2.in‚0 disk.sda.out‚ 873.439453125) proc.memory.availableMB‚ 1398.47265625) disk./.usagePercent‚75.78965879637089 disk.rbd2.out‚2.1875% disk./.usageGB‚2710.6053047180176" disk.nvme0n1.out‚ 81455.0859375# disk.nvme1n1.out‚35736.89453125 disk.md0.in‚ 4.1796875 disk.md0.out‚ 840.23828125 disk.sdb2.in‚ 0.6328125 disk.sda.in‚3.546875 disk.sda1.out‚0 disk.sda2.out‚ 873.439453125 disk.nvme1n1.in‚ 744.32421875 disk.rbd0.out‚2.1875 disk.sdb1.out‚0! disk.dm-0.out‚117191.98046875 cpu‚25.17673281812572 proc.cpu.threads‚17 disk.sda2.in‚3.546875 disk.dm-0.in‚ 1683.86328125[6¡€“ù‚(:ˆ ëÍéÔâÿˆn gpu.0.gpu‚19.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.715' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.17875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ǪWÊ2ú- _wandbruntime‚7846  _runtime‚7846“´Kçû€N C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚571716376108032 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169617283 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚622347739328512 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚622347715388416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7642174311894 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 309595175 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 124491895 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚571717304095744 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚571717304095744 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚564045518613504 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 309595175 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 139977892 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83866643 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7670857494528 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169643653 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124491275 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40624632 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 169617283 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 % _timestamp‚1.7885037882589247e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚564045536616448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7670857494528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83866645 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitorœç;.on_train_batch_end/active_bytes.large_pool.freed‚564045518613504 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7671767479296 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139977894 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 309621547 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚564045536616448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309621547 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8014173801472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139977894 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40625250 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚553046058114594 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 trainer/global_step‚6699 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139977894 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40625250 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚553046058114594 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 trainer/global_step‚6699 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 309595175 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 124491895 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚571717304095744 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚571717304095744 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚564045518613504 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 309595175 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 139977892 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83866643 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7670857494528 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169643653 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124491275 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40624632 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 169617283 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 % _timestamp‚1.7885037882589247e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚564045536616448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7670857494528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83866645 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚564045518613504 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7671767479296 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139977894 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 309621547 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚564045536616448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309621547 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8014173801472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B¯I®Ç‡Á % _timestamp‚1.7885038444399264e+09 _runtime‚7902.218074093 _step‚551 & lr-AdamW/pg1‚0.0005110629812036212 & lr-AdamW/pg2‚0.0005110629812036212 trainer/global_step‚6749­S78ÌˆÆ & lr-AdamW/pg1‚0.0005110629812036212 & lr-AdamW/pg2‚0.0005110629812036212 trainer/global_step‚6749 % _timestamp‚1.7885038444399264e+09 _runtime‚7902.218074093 _step‚551§–È]é2‰- _wandbruntime‚7902  _runtime‚7902…o6û'ŠõO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7725823459328 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 170863698 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 125367862 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚7902.220925706 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84457828 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚568044853818880 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚568044871821824 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40910032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚575770677278208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 trainer/global_step‚6749 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 311803086 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7697842268728 $ _timestamp‚1.788503844443162e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚568044871821824 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚575771605252608 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7725823459328 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8071998580736 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140965732 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚626768774906368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚568044853818880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚575770677278208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140965732 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8071763507200 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚556972680228872 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚549274837960144bVŒÐ‹úO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚568044871821824 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 A8DeviceStatsMonitor.on_train_batch_start/num_ddÄc0&evice_alloc‚1185 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚575771605252608 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7725823459328 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8071998580736 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140965732 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚626768774906368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚568044853818880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚575770677278208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140965732 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8071763507200 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚556972680228872 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚549274837960144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7725823459328 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 170863698 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 125367862 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚7902.220925706 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84457828 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚568044853818880 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚568044871821824 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40910032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚575770677278208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 trainer/global_step‚6749 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 311803086 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7697842268728 $ _timestamp‚1.788503844443162e+09¨ÚZ–$2Œ- _wandbruntime‚7903  _runtime‚7903¦úµ'€N UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 170888555 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7727854904320 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 _step‚553 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚575853417824256 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚557051890082330 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125385109 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚557052815096902 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚575854345811968 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 125385729 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚626859000610816 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885038454784575e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140985894 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40915216 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚626859024550912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚568126490907648 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 311874449 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚634932181404672 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚568126490907648 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 311874449 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚634932181404672 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125385109 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚557052815096902 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚575854345811968 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 125385729 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚626859000610816 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885038454784575e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140985894 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40915216 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚626859024550912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 % _timestamp‚1.7885039018365035e+09 _step‚556 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚572125808110080 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7781910884352 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚572125808110080 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 172082256 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚553225717687344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚560979437211180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚579907718994432 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8130403231232 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚639 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 172082256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7781910884352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 269499392 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 172108600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚579907718994432 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7782820855808 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚579908646968832 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚639410463360000 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 126262295 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141973732 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚579908646968832 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚639410732859392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8130648790528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 172108600 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚572125826113024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85061078 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚639 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 172082256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7781910884352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 269499392 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 172108600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚579907718994432 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7782820855808 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚579908646968832 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚639410463360000 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 126262295 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141973732 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚579908646968832 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚639410732859392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8130648790528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 172108600 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚572125826113024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85061078 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 % _timestamp‚1.7885039018365035e+09¬ÀK©2 - _wandbruntime‚7960  _runtime‚7960ʦü¡€N ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚553304718242528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚631370285833216 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 172107087 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7753928822110 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _step‚557 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 172133457 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 % _timestamp‚1.7885039030049033e+09 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 314127351 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚579991387528192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚579990459540480 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141993894 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚561059572079210 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚553304735281888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7783032344576 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7783032344576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚7960.783820967 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7783942329344 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126279548 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚579991387528192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚572207445198848 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚572207427195904 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172107087 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚650 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141993892 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85073143 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7754836797322 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚561058647064638 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 314100979 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚572207445198848 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7783942329344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141993892 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006L“Š'¢…N N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85073143 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7754836797322 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚561058647064638 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 314100979 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚572207445198848 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7783942329344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141993892 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚553304718242528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚631370285833216 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 172107087 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7753928822110 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _step‚557 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 172133457 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 % _timestamp‚1.7885039030049033e+09 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 314127351 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚579991387528192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚579990459540480 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141993894 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚561059572079210 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚553304735281888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7783032344576 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7783032344576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚7960.783820967 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7783942329344 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126279548 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚579991387528192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚572207445198848 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚572207427195904 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172107087 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚650 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141993892­´öÒã2£- _wandbruntime‚7960  _runtime‚7960t ƒX¤Ò 0loss/FRt2/assignment_loss‚2.9699659744898477 2loss/SRqqt4/assignment_loss‚1.3839048560708762 1loss/FBt4/assignment_loss‚0.10144801003751683 .loss/FRt3/detection_loss‚2.028546413808477 1loss/SRqqt2/detection_loss‚2.6416177309077717 .loss/FBt1/detection_loss‚0.729258230559423 0loss/FRt3/assignment_loss‚1.9268101594027351 _step‚558 .loss/FRt4/assignment_loss‚3.05518359487707 .loss/FRt2/detection_loss‚2.301319625801765 1loss/SRqqt1/detection_loss‚1.8643472970408552 .loss/FBt3/detection_loss‚2.111675903201103 2loss/SRqqt3/assignment_loss‚1.0302141742719877 1loss/SRqqt3/detection_loss‚1.0677470712635284 0loss/SRqqt2/assignment_loss‚1.27025775157887 /loss/FRt4/detection_loss‚3.4531388824636284 .loss/FBt4/detection_loss‚2.315860072771708 &loss/total_loss‚41.731938343846025 1loss/SRqqt1/assignment_loss‚1.552187690840048 1loss/FBt1/assignment_loss‚0.09277480779714814 1loss/FBt2/assignment_loss‚0.10252430478143044 2loss/FBt3/assignment_loss‚0.017193906829925255 .loss/FRt1/detection_loss‚1.716274053780944 0loss/SRqqt4/detection_loss‚2.511345104376475 epoch‚5 trainer/global_step‚6799 0loss/FRt1/assignment_loss‚2.8702336894141305 .loss/FBt2/detection_loss‚2.618109037478765 % _timestamp‚1.7885039030081108e+09 _runtime‚7960.785139743¹ ]¥× 0loss/FRt2/assignment_loss‚2.9699659744898477 2loss/SRqqt4/assignment_loss‚1.3839048560708762 1loss/FBt4/assignment_loss‚0.10144801003751683 .loss/FRt3/detection_loss‚2.028546413808477 1loss/SRqqt2/detection_loss‚2.6416177309077717 .loss/FBt1/detection_loss‚0.729258230559423 0loss/FRt3/assignment_loss‚1.9268101594027351 _step‚558 .loss/FRt4/assignment_loss‚3.05518359487707 .loss/FRt2/detection_loss‚2.301319625801765 1loss/SRqqt1/detection_loss‚1.8643472970408552 .loss/FBt3/detection_loss‚2.111675903201103 2loss/SRqqt3/assignment_loss‚1.0302141742719877 1loss/SRqqt3/detection_loss‚1.0677470712635284 0loss/SRqqt2/assignment_loss‚1.27025775157887 /loss/FRt4/detection_loss‚3.4531388824636284 .loss/FBt4/detection_loss‚2.315860072771708 &loss/total_loss‚41.731938343846025 1loss/SRqqt1/assignment_loss‚1.552187690840048 1loss/FBt1/assignment_loss‚0.09277480779714814 1loss/FBt2/assignment_loss‚0.10252430478143044 2loss/FBt3/assignment_loss‚0.017193906829925255 .loss/FRt1/detection_loss‚1.716274053780944 0loss/SRqqt4/detection_loss‚2.511345104376475 epoch‚5 trainer/global_step‚6799 0loss/FRt1/assignment_loss‚2.8702336894141305 .loss/FBt2/detection_loss‚2.618109037478765 % _timestamp‚1.7885039030081108e+09 _runtime‚7960.785139743®ÆÝ  “¦‚(:ˆ ãÎéÔ㧊n gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚172.395' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.09875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÇÚMj§‚(:ß ãÎéÔÇúËn disk.sdb1.in‚0 disk.nvme0n1.in‚ 939.5390625 disk.nvme1n1.in‚ 744.32421875) proc.memory.availableMB‚ 1398.08984375 network.recv‚ 2.628444e+06 disk.rbd2.in‚0 disk.md0p1.in‚ 4.18359375 disk.md0p1.out‚ 853.9140625 network.sent‚ 1.7455324e+07) disk./.usagePercent‚75.78977558965356 disk.sdb2.in‚ 0.6328125 disk.sda.in‚ 3.55078125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 1683.86328125 disk.sdb.in‚ 0.6328125# disk.nvme1n1.out‚35738.89453125 disk.md0.out‚ 853.9140625 proc.cpu.threads‚17 disk.sdb.out‚887.65087890625 disk.sda2.in‚ 3.55078125 disk.nvme0n1.out‚ 83283.71875! proc.memory.rssMB‚ 2959.484375 cpu‚25.174580359892673 disk.rbd0.out‚2.1875! disk.dm-0.out‚119022.61328125 disk.md0.in‚ 4.18359375% disk./.usageGB‚2710.6094818115234! disk.sdb2.out‚887.65087890625! disk.sda2.out‚887.65087890625$ memory_percent‚91.46673679351807) proc.memory.percent‚18.06325912475586 disk.rbd2.out‚2.1875 disk.sda.out‚887.65087890625 disk.sda1.out‚0}6¹%‘¨‚(:† òÎéÔÔÇ¡m gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡ï f©‚(:Û òÎéÔÃöám disk.sda2.in‚ 3.55078125 disk.nvme1n1.in‚ 744.32421875" disk.nvme1n1.out‚ 35739.1484375 disk.sda.out‚889.25244140625! proc.memory.rssMB‚ 2959.484375 proc.cpu.threads‚17! disk.sda2.out‚889.25244140625 network.sent‚ 1.7563164e+07 disk.rbd2.in‚0! disk.dm-0.out‚119230.18359375# disk.nvme0n1.out‚83491.03515625 disk.sdb2.in‚ 0.6328125! disk.sdb2.out‚889.25244140625 disk.sda1.in‚0 disk.sdb1.in‚0 disk.md0.in‚ 4.18359375 disk.md0.out‚855.4375 disk.rbd0.out‚2.1875$ memory_percent‚91.46687984466553 cpu‚25.174321174680728 disk.sdb.out‚889.25244140625 disk.nvme0n1.in‚ 939.5390625 disk.sda.in‚ 3.55078125 disk.sdb1.out‚0 network.recv‚ 2.634332e+06) disk./.usagePercent‚75.78835284481558 disk.md0p1.in‚ 4.18359375 disk.md0p1.out‚855.4375 disk.rbd0.in‚0) proc.memory.percent‚18.06325912475586% disk./.usageGB‚2710.5585975646973 disk.dm-0.in‚ 1683.86328125 disk.sdb.in‚ 0.6328125 disk.rbd2.out‚2.1875 disk.sda1.out‚0) proc.memory.availableMB‚ 1398.06640625]vÞˆ“ª‚(:ˆ ÏéԦЂm gpu.0.gpu‚19.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚119.091' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.77275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0PwÖ_c«‚(:Ø ÏéÔÊ­Šn disk.sdb1.out‚0 disk.dm-0.in‚ 1683.86328125 disk.nvme0n1.in‚ 939.5390625 disk.rbd2.in‚0 disk.rbd2.out‚2.1875 disk.sdb.in‚ 0.6328125 disk.sda1.in‚0$ memory_percent‚91.46590232849121) proc.memory.percent‚18.06325912475586 network.sent‚ 1.7570716e+07 disk.rbd0.in‚0$ disk./.usageGB‚2710.637264251709 disk.rbd0.out‚2.1875 disk.md0p1.in‚ 4.18359375 disk.md0p1.out‚ 857.0390625 disk.sda2.out‚890.9189453125# disk.nvme1n1.out‚35820.03515625 proc.cpu.threads‚17 disk.sdb2.in‚ 0.6328125 disk.sda.out‚890.9189453125' disk./.usagePercent‚75.790552398309 disk.dm-0.out‚119539.3046875# disk.nvme0n1.out‚83719.26953125 disk.md0.out‚ 857.0390625 cpu‚25.17405543787634 network.recv‚ 2.637828e+06 disk.md0.in‚ 4.18359375( proc.memory.availableMB‚ 1398.2265625! proc.memory.rssMB‚ 2959.484375 disk.sdb.out‚890.9189453125 disk.sda2.in‚ 3.55078125 disk.nvme1n1.in‚ 744.32421875 disk.sda1.out‚0 disk.sdb2.out‚890.9189453125 disk.sda.in‚ 3.55078125 disk.sdb1.in‚0Ê\òa“¬‚(:ˆ ÏéÔ…Ž—m gpu.0.gpu‚15.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚155.961' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.99025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gñEa­‚(:Ö ÏéÔéªüm disk.sda.in‚ 3.55078125 disk.rbd0.in‚0# memory_percent‚91.4651870727539! disk.sdb2.out‚892.71044921875 disk.sdb1.out‚0! disk.dm-0.out‚119657.62890625 disk.sda2.in‚ 3.55078125 cpu‚25.173762333412327 network.sent‚ 1.7576752e+07 network.recv‚ 2.642223e+06( disk./.usagePercent‚75.7914907976706 disk.sda1.in‚0 disk.rbd2.out‚2.1875 proc.cpu.threads‚17$ disk./.usageGB‚2710.670825958252 disk.nvme1n1.in‚ 744.32421875 disk.md0.in‚ 4.18359375& proc.memory.availableMB‚ 1398.34375) proc.memory.percent‚18.06325912475586 disk.sdb1.in‚0 disk.dm-0.in‚ 1683.86328125! disk.sda2.out‚892.71044921875 disk.rbd2.in‚0 disk.nvme1n1.out‚ 35851.9375 disk.sdb2.in‚ 0.6328125 disk.sda1.out‚0 disk.rbd0.out‚2.1875# disk.nvme0n1.out‚83805.69140625 disk.sda.out‚892.71044921875 disk.md0p1.in‚ 4.18359375 disk.md0p1.out‚ 858.765625 disk.sdb.out‚892.71044921875 disk.nvme0n1.in‚ 939.5390625 disk.md0.out‚ 858.765625! proc.memory.rssMB‚ 2959.484375 disk.sdb.in‚ 0.6328125“ÁK2®- _wandbruntime‚8017  _runtime‚8017l̶‹Ç¯Á & lr-AdamW/pg2‚0.0004958857332816255 trainer/global_step‚6849 % _timestamp‚1.7885039592458162e+09 _runtime‚8017.023174774 _step‚559 & lr-AdamW/pg1‚0.0004958857332816255k]¨Ì°Æ & lr-AdamW/pg1‚0.0004958857332816255 & lr-AdamW/pg2‚0.0004958857332816255 trainer/global_step‚6849 % _timestamp‚1.7885039592458162e+09 _runtime‚8017.023174774 _step‚559¯ãóh2±- _wandbruntime‚8017  _runtime‚80174ã±"²öO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚6849 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 173353502 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚576206780404224 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚643980448290304 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212004864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚576206762401280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚557176597414544 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚615 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚576206762401280 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7838908280832 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235944960 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7808688803940 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚635791345351168 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7809596778944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41492400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 173327158 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚584044760710656 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚576206780404224 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 127156113 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142981732 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 173327158 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚635791369291264 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 173353502 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 316335236 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚584045688685056 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚584045688685056 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.cu†ÌÉóKrrent‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85664328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚584044760710656 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7837998309376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8188866994176 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚564986194193488 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚564985269179124 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚557176580375184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185„ü!(³ûO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚584044760710656 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7837998309376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8188866994176 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚564986194193488 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚564985269179124 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚557176580375184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚6849 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 173353502 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚576206780404224 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚643980448290304 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212004864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚576206762401280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚557176597414544 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚615 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚576206762401280 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7838908280832 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235944960 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7808688803940 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚635791345351168 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7809596778944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41492400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 173327158 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚584044760710656 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚576206780404224 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 127156113 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142981732 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 173327158 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚635791369291264 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 173353502 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 316335236 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚584045688685056 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚584045688685056 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85664328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 173378359 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚576288399490048 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚576288399490048 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚557255597969728 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7840029754368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85676395 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 316353881 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7810714052430 _step‚561 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 316380253 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚584127501256704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7840029754368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41498223 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 143001894 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 % _timestamp‚1.7885039602136703e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7839119769600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85676393 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 316353881 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143001894 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127174618 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚584128429244416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8190263449600 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127173996 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 173351989 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚576288381487104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚576288381487104 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚565066329061518 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _runtime‚8017.991411242 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚6849 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚644071612027904 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚635881594995712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144ÃÕ= '¶…N \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7810714052430 _step‚561 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 316380253 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚584127501256704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7840029754368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41498223 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 143001894 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 % _timestamp‚1.7885039602136703e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7839119769600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85676393 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 316353881 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143001894 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127174618 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚584128429244416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8190263449600 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127173996 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 173351989 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚576288381487104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚576288381487104 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚565066329061518 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _runtime‚8017.991411242 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚6849 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚644071612027904 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚635881594995712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143001892 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚584128429244416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8190040972288 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 173378359 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚576288399490048 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚576288399490048 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚557255597969728 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7840029754368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85676395 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 316353881 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0±x Áý2·- _wandbruntime‚8017  _runtime‚8017L1]¸Ð /loss/FRt1/detection_loss‚1.5208547488904978 /loss/FBt4/detection_loss‚2.2877600312232973 _runtime‚ 8017.9931929 1loss/SRqqt2/assignment_loss‚1.473440118772643 1loss/FBt1/assignment_loss‚0.09729547785575768 1loss/FBt2/assignment_loss‚0.22456488702997981 0loss/FBt4/assignment_loss‚0.5775012566494298 /loss/FRt1/assignment_loss‚2.124916717117908 0loss/FBt3/assignment_loss‚0.2136783801496759 /loss/FRt2/detection_loss‚2.4141416715049164 /loss/SRqqt4/detection_loss‚2.53911239374429 _step‚562 0loss/FRt4/assignment_loss‚3.4360416951002897 2loss/SRqqt3/assignment_loss‚0.9311352724253776 0loss/SRqqt1/detection_loss‚1.871966337716138 1loss/SRqqt3/detection_loss‚1.1590093021946295 %loss/total_loss‚41.18391135346565 epoch‚5 /loss/FRt4/detection_loss‚3.1542292568418713 1loss/SRqqt2/detection_loss‚2.2034357224191936 /loss/FBt2/detection_loss‚2.6665870398283005 0loss/FRt3/assignment_loss‚1.7402351754052299 /loss/FBt1/detection_loss‚0.57985U»ïcH03858252213 .loss/FBt3/detection_loss‚2.175387209111994 1loss/SRqqt1/assignment_loss‚1.018333665681658 /loss/FRt3/detection_loss‚1.9744758367804545 trainer/global_step‚6849 % _timestamp‚1.7885039602161968e+09 /loss/FRt2/assignment_loss‚3.261022701496031 1loss/SRqqt4/assignment_loss‚1.538936069700867*l U[¹Õ 0loss/FBt3/assignment_loss‚0.2136783801496759 /loss/FRt2/detection_loss‚2.4141416715049164 /loss/SRqqt4/detection_loss‚2.53911239374429 _step‚562 0loss/FRt4/assignment_loss‚3.4360416951002897 2loss/SRqqt3/assignment_loss‚0.9311352724253776 0loss/SRqqt1/detection_loss‚1.871966337716138 1loss/SRqqt3/detection_loss‚1.1590093021946295 %loss/total_loss‚41.18391135346565 epoch‚5 /loss/FRt4/detection_loss‚3.1542292568418713 1loss/SRqqt2/detection_loss‚2.2034357224191936 /loss/FBt2/detection_loss‚2.6665870398283005 0loss/FRt3/assignment_loss‚1.7402351754052299 /loss/FBt1/detection_loss‚0.5798503858252213 .loss/FBt3/detection_loss‚2.175387209111994 1loss/SRqqt1/assignment_loss‚1.018333665681658 /loss/FRt3/detection_loss‚1.9744758367804545 trainer/global_step‚6849 % _timestamp‚1.7885039602161968e+09 /loss/FRt2/assignment_loss‚3.261022701496031 1loss/SRqqt4/assignment_loss‚1.538936069700867 /loss/FRt1/detection_loss‚1.5208547488904978 /loss/FBt4/detection_loss‚2.2877600312232973 _runtime‚ 8017.9931929 1loss/SRqqt2/assignment_loss‚1.473440118772643 1loss/FBt1/assignment_loss‚0.09729547785575768 1loss/FBt2/assignment_loss‚0.22456488702997981 0loss/FBt4/assignment_loss‚0.5775012566494298 /loss/FRt1/assignment_loss‚2.124916717117908²­Iù ‘º‚(:† ŸÏéÔ˃¶n gpu.0.gpu‚52.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚168.992' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âidc»‚(:Ø ŸÏéÔì€ön cpu‚25.173525494129954 proc.cpu.threads‚17 network.sent‚ 1.7593197e+07) disk./.usagePercent‚75.79760724588512 disk.sda1.out‚0 disk.md0.in‚ 4.18359375 disk.rbd2.out‚2.1875( proc.memory.availableMB‚ 1398.1796875 disk.rbd0.in‚0 disk.sdb1.out‚0! disk.sda2.out‚894.65576171875 disk.rbd2.in‚0) proc.memory.percent‚18.06325912475586$ disk./.usageGB‚2710.889579772949$ memory_percent‚91.46618843078613! proc.memory.rssMB‚ 2959.484375 disk.dm-0.out‚ 119927.671875 disk.sdb.in‚ 0.6328125 disk.sdb.out‚894.65576171875 disk.sda2.in‚ 3.55078125! disk.nvme1n1.out‚ 35964.421875 network.recv‚ 2.646071e+06 disk.sda.in‚ 3.55078125 disk.dm-0.in‚ 1683.86328125 disk.nvme0n1.out‚83963.25! disk.sdb2.out‚894.65576171875 disk.sda1.in‚0 disk.sdb1.in‚0 disk.md0p1.out‚ 860.63671875 disk.nvme1n1.in‚ 744.32421875 disk.sdb2.in‚ 0.6328125 disk.rbd0.out‚2.1875 disk.md0p1.in‚ 4.18359375 disk.md0.out‚ 860.63671875 disk.sda.out‚894.65576171875 disk.nvme0n1.in‚ 939.5390625`ØÁ.“¼‚(:ˆ ®ÏéÔ‹ûùl gpu.0.gpu‚20.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚132.399' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.09975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I6Gj½‚(:ß ®ÏéÔŠØm$ disk./.usageGB‚2711.140598297119$ memory_percent‚91.46618843078613 disk.sdb2.in‚ 0.6328125 disk.sda.out‚896.16552734375 disk.sdb1.out‚0 disk.md0p1.in‚ 4.18359375 disk.nvme1n1.in‚ 744.32421875 disk.md0.out‚ 862.07421875 disk.sda1.in‚0 disk.rbd0.out‚2.1875 disk.dm-0.out‚ 120119.53125 network.recv‚ 2.651705e+06) disk./.usagePercent‚75.80462582888032# disk.nvme0n1.out‚84120.48828125 disk.sda.in‚ 3.55078125# disk.nvme1n1.out‚35999.04296875 disk.rbd0.in‚0! proc.memory.rssMB‚ 2959.484375) proc.memory.percent‚18.06325912475586 cpu‚25.173272809443603! disk.sda2.out‚896.16552734375 disk.nvme0n1.in‚ 939.5390625 disk.rbd2.in‚0 disk.md0p1.out‚ 862.07421875 proc.cpu.threads‚17 disk.sdb.out‚896.16552734375 disk.md0.in‚ 4.18359375 disk.rbd2.out‚2.1875 disk.sdb1.in‚0 disk.sdb.in‚ 0.6328125 disk.sda2.in‚ 3.55078125! disk.sdb2.out‚896.16552734375 disk.sda1.out‚0 disk.dm-0.in‚ 1683.86328125( proc.memory.availableMB‚ 1398.1796875 network.sent‚ 1.7690405e+07X„𑾂(:† ½ÏéÔÈõÏm gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.916' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.229 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—«•Bc¿‚(:Ø ½ÏéÔ‚Üým) disk./.usagePercent‚75.80462582888032$ disk./.usageGB‚2711.140598297119 disk.sda1.out‚0 disk.rbd0.out‚2.1875 disk.sdb1.in‚0 disk.sdb.in‚ 0.6328125 disk.sda2.in‚ 3.55078125 disk.rbd2.out‚2.1875 cpu‚25.172980150111037 disk.sda.out‚898.18017578125 disk.md0p1.out‚ 864.03125 disk.sdb.out‚898.18017578125 disk.md0.in‚ 4.18359375# disk.nvme1n1.out‚35999.37890625 network.recv‚ 2.654473e+06 disk.rbd0.in‚0 disk.dm-0.in‚ 1683.86328125! disk.dm-0.out‚120120.00390625 disk.nvme0n1.in‚ 939.5390625 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.sdb2.in‚ 0.6328125 disk.sda1.in‚0 disk.md0p1.in‚ 4.18359375 disk.md0.out‚ 864.03125! disk.sdb2.out‚898.18017578125 disk.nvme1n1.in‚ 744.32421875 disk.nvme0n1.out‚ 84120.625 disk.sda.in‚ 3.55078125 disk.sdb1.out‚0! disk.sda2.out‚898.18017578125$ memory_percent‚91.46616458892822) proc.memory.availableMB‚ 1398.18359375! proc.memory.rssMB‚ 2959.484375) proc.memory.percent‚18.06325912475586 network.sent‚ 1.7692584e+07$¿Øð“À‚(:ˆ ÌÏéÔ‰–Ëm gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚181.699' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.42475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ò@_{YÁ‚(:Î ÌÏéÔÀÔˆn disk.sdb.out‚ 899.4765625 disk.nvme1n1.in‚ 744.32421875 disk.md0.in‚ 4.18359375 disk.md0.out‚ 865.27734375 disk.rbd2.out‚2.1875 disk.dm-0.in‚ 1683.86328125# disk.nvme0n1.out‚84487.50390625 disk.nvme1n1.out‚ 36233.90625 disk.sdb2.in‚ 0.6328125$ memory_percent‚91.46542549133301 disk.sda2.out‚ 899.4765625 disk.sda1.in‚0 disk.rbd0.out‚2.1875 disk.sdb1.in‚0 disk.sda.in‚ 3.55078125 cpu‚25.172751763096645 proc.cpu.threads‚17 disk.md0p1.out‚ 865.27734375 network.recv‚ 2.658919e+06 disk.sdb.in‚ 0.6328125 disk.rbd2.in‚0 disk.sdb2.out‚ 899.4765625 disk.md0p1.in‚ 4.18359375$ disk./.usageGB‚2711.139579772949 disk.sda2.in‚ 3.55078125 disk.nvme0n1.in‚ 939.5390625! proc.memory.rssMB‚ 2959.484375) proc.memory.percent‚18.06325912475586 network.sent‚ 1.770078e+07 disk.rbd0.in‚0 disk.sdb1.out‚0! disk.dm-0.out‚120721.41015625 disk.sda.out‚ 899.4765625( proc.memory.availableMB‚ 1398.3046875 disk.sda1.out‚0) disk./.usagePercent‚75.80459735051825%3pÉ2Â- _wandbruntime‚8074  _runtime‚8074vŠ(…ÇÃÁ _runtime‚8074.475719801 _step‚563 & lr-AdamW/pg1‚0.0004883057877787293 & lr-AdamW/pg2‚0.0004883057877787293 trainer/global_step‚6899 % _timestamp‚1.7885040166984053e+09f¨ÌÄÆ & lr-AdamW/pg1‚0.0004883057877787293 & lr-AdamW/pg2‚0.0004883057877787293 trainer/global_step‚6899 % _timestamp‚1.7885040166984053e+09 _runtime‚8074.475719801 _step‚563³ÞòÓ2Å- _wandbruntime‚8074  _runtime‚8074VRoÿû'ÆõO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143989734 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚561127477141744 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 318561792 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚580287734695424 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 128050487 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚648550056214016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚640302630573568 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚580287734695424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚622 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86267578 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚588181802426880 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 174598404 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚648550311033344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7864566059048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7894995705856 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚588182730401280 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚6899 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7865474034052 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚640302654513664 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚580287716692480 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86267580 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 318588138 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174572060 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚561127460102384 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7894085734400 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚568992026161432 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚8074.477160448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41782287 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8247425640448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚620 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚588182730401280 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚588181802426880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.788504016699695e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7894995705856 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143989732 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚568992951175796 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 128049865 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41782907 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615ÉÁØŽ(ÇúO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.788504016699695e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7894995705856 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143989732 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚568992951175796 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 128049865 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41782907 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143989734 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚561127477141744 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 318561792 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚580287734695424 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 128050487 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚648550056214016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚640302630573568 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚580287734695424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚622 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86267578 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚588181802426880 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 174598404 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚648550311033344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7864566059048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7894995705856 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚588182730401280 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚6899 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7865474034052 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚640302654513664 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚580287716692480 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86267580 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 318588138 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174572060 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚561127460102384 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7894085734400 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚568992026161432 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚8074.477160448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41782287 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8247425640448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚620 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚588182730401280 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚588181802426880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318633155 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚580369335778304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128067659 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227543040 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚588264542972928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8248841297920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144009892 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174623261 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚648641493972992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174596891 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41788637 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7865683332326 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 128068282 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7895207194624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144009892 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 318606783 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174623261 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41788016 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 203602944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8248637694976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7896117179392 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚640392880218112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 174596891 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 318633155 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚580369335778304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚561206494736288 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7866591307538 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 % _timestamp‚1.7885040177285864e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚569072161029254 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚561206477696928 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚565 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 318606783 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚6899˜Có± 'Ê…N H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 318606783 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚6899 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚648641721516032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318633155 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚580369335778304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128067659 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227543040 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚588264542972928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8248841297920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144009892 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174623261 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚648641493972992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174596891 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41788637 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7865683332326 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 128068282 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7895207194624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144009892 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 318606783 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174623261 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41788016 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 203602944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8248637694976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7896117179392 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚640392880218112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 174596891 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 318633155 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚580369335778304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚561206494736288 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7866591307538 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 % _timestamp‚1.7885040177285864e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚569072161029254 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚561206477696928 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚565µøz¯õ2Ë- _wandbruntime‚8075  _runtime‚8075âȤ¶ZÌÔ /loss/FBt1/detection_loss‚0.5186541478385703 _step‚566 2loss/SRqqt4/assignment_loss‚1.3608686751127244 0loss/FBt4/assignment_loss‚0.2211016881838077 .loss/FRt1/assignment_loss‚2.28888124498454 0loss/FBt2/assignment_loss‚0.5435903056702955 1loss/SRqqt2/detection_loss‚2.3309796584977045 &loss/total_loss‚42.388431647403074 2loss/SRqqt1/assignment_loss‚1.6649362943295776 /loss/FRt1/detection_loss‚1.7987230889433372 1loss/SRqqt4/detection_loss‚2.6624819898605345 /loss/FRt2/assignment_loss‚3.142256776069073 0loss/FRt3/assignment_loss‚1.2554691504029667 2loss/SRqqt2/assignment_loss‚1.8455051448610094 .loss/FRt2/detection_loss‚2.217762419359481 1loss/SRqqt3/detection_loss‚1.0591050781705211 .loss/FBt4/detection_loss‚2.397139448672533 1loss/FBt1/assignment_loss‚0.13370821166437927 0loss/FBt3/assignment_loss‚0.2820744881406426 /loss/FRt3/detection_loss‚1.8317108240489866 /loss/FRt4/detection_loss‚2.8764279232575345 /loss/FBt2/detection_loss‚2.9172901570796967 trainer/global_step‚6899 % _timestamp‚1.7885040177316408e+09 _runtime‚8075.508665195 /loss/FRt4/assignment_loss‚3.071728149285683 .loss/FBt3/detection_loss‚2.578392279148102 epoch‚5 2loss/SRqqt3/assignment_loss‚1.0436615206268245 1loss/SRqqt1/detection_loss‚2.3459829831945487ç¦ù6_ÍÙ 2loss/SRqqt4/assignment_loss‚1.3608686751127244 0loss/FBt4/assignment_loss‚0.2211016881838077 .loss/FRt1/assignment_loss‚2.28888124498454 0loss/FBt2/assignment_loss‚0.5435903056702955 1loss/SRqqt2/detection_loss‚2.3309796584977045 &loss/total_loss‚42.388431647403074 2loss/SRqqt1/assignment_loss‚1.6649362943295776 /loss/FRt1/detection_loss‚1.7987230889433372 1loss/SRqqt4/detection_loss‚2.6624819898605345 /loss/FRt2/assignment_loss‚3.142256776069073 0loss/FRt3/assignment_loss‚1.2554691504029667 2loss/SRqqt2/assignment_loss‚1.8455051448610094 .loss/FRt2/detection_loss‚2.217762419359481 1loss/SRqqt3/detection_loss‚1.0591050781705211 .loss/FBt4/detection_loss‚2.397139448672533 1loss/FBt1/assignment_loss‚0.13370821166437927 0loss/FBt3/assignment_loss‚0.2820744881406426 /loss/FRt3/detection_loss‚1.8317108240489866 /loss/FRt4/detection_loss‚2.8764279232575345 /loss/FBt2/detection_loss‚2.9172901570796967 trainer/global_step‚6899 % _timestamp‚1.7885040177316408e+09 _runtime‚8075.508665195 /loss/FRt4/assignment_loss‚3.071728149285683 .loss/FBt3/detection_loss‚2.578392279148102 epoch‚5 2loss/SRqqt3/assignment_loss‚1.0436615206268245 1loss/SRqqt1/detection_loss‚2.3459829831945487 /loss/FBt1/detection_loss‚0.5186541478385703 _step‚566¶…}–g΂(:Ü ÛÏéÔÝôŸn disk.sdb.in‚ 0.6328125 disk.sdb.out‚901.3583984375! disk.nvme0n1.out‚ 84487.609375 disk.sda1.in‚0 disk.md0p1.out‚ 867.1015625 disk.dm-0.out‚120721.7734375* proc.memory.percent‚17.862582206726074 proc.cpu.threads‚17 disk.nvme1n1.in‚ 744.32421875 disk.rbd2.in‚0 disk.sda.in‚ 3.55078125 cpu‚25.172492818651047 network.sent‚ 1.7717261e+07 network.recv‚ 2.662692e+06) disk./.usagePercent‚75.78977718956153 disk.sda1.out‚0% disk./.usageGB‚2710.6095390319824 disk.md0.in‚ 4.18359375$ memory_percent‚91.29519462585449( proc.memory.availableMB‚ 1426.1953125# proc.memory.rssMB‚ 2926.60546875 disk.sda2.in‚ 3.55078125 disk.nvme0n1.in‚ 939.5390625 disk.md0.out‚ 867.1015625 disk.sdb2.in‚ 0.6328125 disk.sdb1.in‚0 disk.md0p1.in‚ 4.18359375 disk.sda2.out‚901.3583984375" disk.nvme1n1.out‚ 36234.1640625 disk.rbd2.out‚2.1875 disk.sdb2.out‚901.3583984375 disk.rbd0.in‚0 disk.rbd0.out‚2.1875 disk.sda.out‚901.3583984375 disk.sdb1.out‚0 disk.dm-0.in‚ 1683.86328125¬ƒúÍ’Ï‚(:‡ ÛÏéÔî”ôù gpu.0.gpu‚31.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.344' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0} S‘Ђ(:† êÏéÔªÆúl gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.676' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.419 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09ü y`Ñ‚(:Õ êÏéÔ¹¢ãm disk.rbd0.in‚0 disk.rbd2.out‚2.1875 proc.cpu.threads‚17( disk./.usagePercent‚75.7897684433979 disk.sdb.out‚ 902.853515625 disk.sda2.in‚ 3.55078125# proc.memory.rssMB‚ 2926.60546875 cpu‚25.172238907507293 network.recv‚ 2.668689e+06 disk.sda1.in‚0 disk.sdb.in‚ 0.6328125 disk.rbd2.in‚0 disk.sdb2.out‚ 902.853515625 disk.sda.in‚ 3.55078125 disk.sdb1.out‚0 disk.md0p1.out‚ 868.5390625 disk.nvme0n1.in‚ 939.5390625* proc.memory.percent‚17.862582206726074% disk./.usageGB‚2710.6092262268066 disk.sda1.out‚0 disk.md0p1.in‚ 4.18359375 disk.dm-0.out‚ 120928.578125 disk.nvme1n1.in‚ 744.32421875 disk.sdb2.in‚ 0.6328125$ memory_percent‚91.29512310028076! disk.nvme0n1.out‚ 84644.859375 disk.nvme1n1.out‚ 36283.71875 disk.sda.out‚ 902.853515625 disk.sdb1.in‚0 disk.dm-0.in‚ 1683.86328125 disk.sda2.out‚ 902.853515625 disk.md0.in‚ 4.18359375 disk.md0.out‚ 868.5390625) proc.memory.availableMB‚ 1426.20703125 network.sent‚ 1.7814574e+07 disk.rbd0.out‚2.1875ž .‘Ò‚(:† ùÏéÔ«½‘m gpu.0.gpu‚50.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚166.548' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.637 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EªÈvgÓ‚(:Ü ùÏéÔ«…šn# disk.nvme1n1.out‚36364.47265625 proc.cpu.threads‚17 disk.md0p1.out‚ 870.54296875 disk.nvme1n1.in‚ 744.32421875 disk.md0.in‚ 4.18359375 cpu‚25.171980191258612 disk.rbd2.out‚2.1875 disk.rbd0.out‚2.1875 disk.sda2.in‚ 3.55078125 disk.nvme0n1.out‚ 84855.15625# memory_percent‚91.2940263748169 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda.in‚ 3.55078125 disk.sda.out‚904.9306640625 disk.sda1.in‚0 disk.sdb1.in‚0 disk.dm-0.in‚ 1683.86328125 disk.md0.out‚ 870.54296875 disk.sdb2.in‚ 0.6328125 disk.sdb2.out‚904.9306640625 disk.sdb.out‚904.9306640625 disk.nvme0n1.in‚ 939.5390625) proc.memory.availableMB‚ 1426.38671875 network.sent‚ 1.781996e+07 network.recv‚ 2.672008e+06 disk.sda1.out‚0 disk.sdb.in‚ 0.6328125# proc.memory.rssMB‚ 2926.60546875* proc.memory.percent‚17.862582206726074( disk./.usagePercent‚75.7905520783274$ disk./.usageGB‚2710.637252807617 disk.sdb1.out‚0 disk.md0p1.in‚ 4.18359375! disk.dm-0.out‚121219.62890625 disk.sda2.out‚904.9306640625=p’Ô‚(:‡ ˆÐéÔÔŠ©m gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.942' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4855 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j©7"jÕ‚(:ß ˆÐéÔ½¦ëm disk.md0.out‚ 871.89453125 disk.rbd2.out‚2.1875 disk.sda.in‚ 3.55078125 disk.rbd0.out‚2.1875 disk.sdb.out‚906.3505859375 disk.sda2.in‚ 3.55078125 disk.nvme0n1.in‚ 939.5390625 disk.rbd2.in‚0 network.recv‚ 2.676429e+06% disk./.usageGB‚2710.6396713256836 disk.sda.out‚906.3505859375 disk.sda1.out‚0# memory_percent‚91.2940263748169 disk.sdb1.out‚0 disk.md0p1.in‚ 4.18359375 disk.nvme1n1.in‚ 744.32421875 disk.sdb2.in‚ 0.6328125 disk.sda1.in‚0 cpu‚25.171696884394873 network.sent‚ 1.7823693e+07 disk.sdb.in‚ 0.6328125 disk.md0.in‚ 4.18359375) proc.memory.availableMB‚ 1426.38671875# proc.memory.rssMB‚ 2926.60546875 disk.dm-0.in‚ 1683.86328125 disk.rbd0.in‚0 disk.nvme0n1.out‚ 84855.15625* proc.memory.percent‚17.862582206726074 proc.cpu.threads‚17) disk./.usagePercent‚75.79061970110476 disk.sdb2.out‚906.3505859375 disk.sdb1.in‚0 disk.md0p1.out‚ 871.89453125! disk.dm-0.out‚121219.76953125 disk.sda2.out‚906.3505859375# disk.nvme1n1.out‚36364.61328125²£:2Ö- _wandbruntime‚8131  _runtime‚8131}.jÆ×À & lr-AdamW/pg2‚0.0004807337345521236 trainer/global_step‚6949 $ _timestamp‚1.788504074026216e+09 _runtime‚8131.803597856 _step‚567 & lr-AdamW/pg1‚0.0004807337345521236«;¥ËØÅ trainer/global_step‚6949 $ _timestamp‚1.788504074026216e+09 _runtime‚8131.803597856 _step‚567 & lr-AdamW/pg1‚0.0004807337345521236 & lr-AdamW/pg2‚0.0004807337345521236·ê.2Ù- _wandbruntime‚8131  _runtime‚8131ß!¾ÚöO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚644813915795968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 241364992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 175843306 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚584368670983680 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7921351289160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7950173159424 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7950173159424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚568 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144997734 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚565078356868944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚657 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚6949 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚592318844143104 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175816962 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7951083130880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚644813939736064 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚572999708158104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚592319772117504 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 128943958 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144997734 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚592318844143104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 175816962 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7920443314156 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 265305088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚653119705656320 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚565078339829584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚584368688986624 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42073787 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 128944617 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86870828 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144997732 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231Œ±(ÛûO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚6949 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚592318844143104 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175816962 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7951083130880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚644813939736064 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚572999708158104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚592319772117504 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 128943958 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144997734 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚592318844143104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 175816962 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7920443314156 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 265305088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚653119705656320 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚565078339829584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚584368688986624 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42073787 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 128944617 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86870828 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144997732 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚644813915795968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 241364992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 175843306 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚584368670983680 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7921351289160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7950173159424 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7950173159424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚568 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144997734 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚565078356868944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚657 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640¸›%h2Ü- _wandbruntime‚8132  _runtime‚8132¡s\A'ÝÿM WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145017892 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚592402512676864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145017892 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚653211362869760 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 % _timestamp‚1.7885040750516095e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8307197429248 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175841793 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 320859685 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7952204604416 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 128962402 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175868163 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚644904141500416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚592401584689152 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175841793 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42078885 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 8132.82956477 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 320886057 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚573078918011562 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145017894 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86882893 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42079507 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8306985437696 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚573079843026134 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚592402512676864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7922468562646 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 320859685 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128961778 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚565157374463488 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚584450308072448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7952204604416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚584450290069504 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86882895 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚622 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 211991552 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7921560587434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 trainer/global_step‚6949 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7951294619648 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 235931648 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚644904165440512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 320886057 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚653211362869760 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 % _timestamp‚1.7885040750516095e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8307197429248 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175841793 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 320859685 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7952204604416 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 128962402 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175868163 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚644904141500416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚592401584689152 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175841793 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42078885 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 8132.82956477 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 320886057 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚573078918011562 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145017894 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86882893 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42079507 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8306985437696 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚573079843026134 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚592402512676864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7922468562646 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 320859685 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128961778 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚565157374463488 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚584450308072448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7952204604416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚584450290069504 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86882895 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚622 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 211991552 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7921560587434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 trainer/global_step‚6949 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7951294619648 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 235931648 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚644904165440512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 320886057 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144¹rì¼"2ß- _wandbruntime‚8132  _runtime‚8132d§9É\àÖ 2loss/SRqqt2/assignment_loss‚1.7300728741288185 1loss/FBt1/assignment_loss‚0.11619135620876832 1loss/FBt3/assignment_loss‚0.19202029701167098 .loss/FRt4/detection_loss‚2.936433027897562 % _timestamp‚1.7885040750541642e+09 1loss/FBt2/assignment_loss‚0.32652729244932743 0loss/FBt4/assignment_loss‚0.2971312384232316 /loss/FBt1/detection_loss‚0.6847324684238085 .loss/FBt4/detection_loss‚2.013114413489466 0loss/FRt4/assignment_loss‚3.5012453070708682 /loss/FRt1/detection_loss‚2.2022899167281316 _runtime‚8132.831166627 _step‚570 0loss/FRt3/assignment_loss‚1.3101285160598108 1loss/SRqqt2/detection_loss‚2.5384014797210694 1loss/SRqqt4/detection_loss‚2.6007405803962187 /loss/FRt2/detection_loss‚2.3331280801185343 .loss/FRt3/detection_loss‚1.654590894989038 1loss/SRqqt3/detection_loss‚0.8091434263003369 /loss/FBt2/detection_loss‚2.5000093206763268 epoch‚5 trainer/global_step‚6949 0loss/FRt1/assignment_loss‚1.9119790619419468 1loss/SRqqt4/assignment_loss‚1.728202965449203 .loss/FBt3/detection_loss‚3.172736326853434 %loss/total_loss‚41.83263880784759 0loss/FRt2/assignment_loss‚3.2638359139131947 2loss/SRqqt1/assignment_loss‚1.2899850703775884 2loss/SRqqt3/assignment_loss‚1.0891651092950876 0loss/SRqqt1/detection_loss‚1.630833869924148ã•û@aáÛ .loss/FRt4/detection_loss‚2.936433027897562 % _timestamp‚1.7885040750541642e+09 1loss/FBt2/assignment_loss‚0.32652729244932743 0loss/FBt4/assignment_loss‚0.2971312384232316 /loss/FBt1/detection_loss‚0.6847324684238085 .loss/FBt4/detection_loss‚2.013114413489466 0loss/FRt4/assignment_loss‚3.5012453070708682 /loss/FRt1/detection_loss‚2.2022899167281316 _runtime‚8132.831166627 _step‚570 0loss/FRt3/assignment_loss‚1.3101285160598108 1loss/SRqqt2/detection_loss‚2.5384014797210694 1loss/SRqqt4/detection_loss‚2.6007405803962187 /loss/FRt2/detection_loss‚2.3331280801185343 .loss/FRt3/detection_loss‚1.654590894989038 1loss/SRqqt3/detection_loss‚0.8091434263003369 /loss/FBt2/detection_loss‚2.5000093206763268 epoch‚5 trainer/global_step‚6949 0loss/FRt1/assignment_loss‚1.9119790619419468 1loss/SRqqt4/assignment_loss‚1.728202965449203 .loss/FBt3/detection_loss‚3.172736326853434 %loss/total_loss‚41.83263880784759 0loss/FRt2/assignment_loss‚3.2638359139131947 2loss/SRqqt1/assignment_loss‚1.2899850703775884 2loss/SRqqt3/assignment_loss‚1.0891651092950876 0loss/SRqqt1/detection_loss‚1.630833869924148 2loss/SRqqt2/assignment_loss‚1.7300728741288185 1loss/FBt1/assignment_loss‚0.11619135620876832 1loss/FBt3/assignment_loss‚0.19202029701167098ºh­l!’â‚(:‡ —ÐéÔ•ën gpu.0.gpu‚52.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.422' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.8555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=×-Ønã‚(:ã —ÐéÔÉïën! disk.sda2.out‚907.99365234375 disk.rbd2.in‚0 disk.rbd2.out‚2.1875! disk.sdb2.out‚907.99365234375 disk.sda.in‚ 3.55078125 disk.sdb1.in‚0 disk.md0.in‚ 4.18359375 disk.md0.out‚ 873.47265625 disk.sdb2.in‚ 0.6328125 disk.sda.out‚907.99365234375 disk.sda1.in‚0# memory_percent‚91.2947416305542 network.sent‚ 1.7842395e+07 disk.rbd0.out‚2.1875 cpu‚25.171436360709286 disk.sdb.in‚ 0.6328125 disk.sda2.in‚ 3.55078125 disk.nvme0n1.out‚ 85222.15625 disk.rbd0.in‚0# proc.memory.rssMB‚ 2926.60546875* proc.memory.percent‚17.862582206726074! disk.dm-0.out‚121677.48046875 disk.sda1.out‚0 proc.cpu.threads‚17 network.recv‚ 2.680545e+06% disk./.usageGB‚2710.7645835876465 disk.nvme1n1.in‚ 744.32421875) disk./.usagePercent‚75.79411230022909 disk.md0p1.in‚ 4.18359375 disk.sdb.out‚907.99365234375 disk.md0p1.out‚ 873.47265625 disk.nvme0n1.in‚ 939.5390625# disk.nvme1n1.out‚36455.32421875 disk.sdb1.out‚0) proc.memory.availableMB‚ 1426.26953125 disk.dm-0.in‚ 1683.86328125¾–O_‘ä‚(:† ¦ÐéÔ•¢ˆm gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚172.884' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.221 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ>_eå‚(:Ú ¦ÐéÔªÎèm proc.cpu.threads‚17 disk.sda.in‚ 3.55078125 disk.sda1.out‚0 disk.md0.in‚ 4.18359375) proc.memory.availableMB‚ 1426.25390625 disk.md0.out‚ 875.31640625% disk./.usageGB‚2710.8895835876465 disk.md0p1.in‚ 4.18359375! disk.dm-0.out‚121952.08203125 cpu‚25.17118986512327# disk.nvme0n1.out‚85379.39453125 disk.nvme1n1.out‚ 36572.6875 disk.md0p1.out‚ 875.31640625 disk.sdb.out‚ 909.912109375$ memory_percent‚91.29483699798584* proc.memory.percent‚17.862582206726074 disk.sda2.in‚ 3.55078125 disk.rbd2.out‚2.1875 disk.sdb2.out‚ 909.912109375 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0# proc.memory.rssMB‚ 2926.60546875 disk.nvme0n1.in‚ 939.5390625 disk.nvme1n1.in‚ 744.32421875 network.sent‚ 1.7939676e+07 network.recv‚ 2.686462e+06 disk.sdb2.in‚ 0.6328125 disk.sda.out‚ 909.912109375 disk.rbd0.out‚2.1875 disk.sdb1.out‚0) disk./.usagePercent‚75.79760735254565 disk.sdb1.in‚0 disk.dm-0.in‚ 1683.86328125 disk.sdb.in‚ 0.6328125 disk.sda2.out‚ 909.912109375ÏžCë“æ‚(:ˆ µÐéÔ»¨Œm gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.383' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.09575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Fdç‚(:Ù µÐéÔ‡çÝm network.sent‚ 1.7944923e+07) disk./.usagePercent‚75.80454988658146 disk.dm-0.in‚ 1683.86328125 disk.sda.out‚911.4462890625 cpu‚25.17093829208678$ disk./.usageGB‚2711.137882232666 disk.md0.out‚ 876.78125 disk.sda1.in‚0 disk.rbd0.out‚2.1875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd0.in‚0 disk.sdb.in‚ 0.6328125 disk.sdb2.out‚911.4462890625 disk.sda1.out‚0 proc.cpu.threads‚17 disk.md0p1.in‚ 4.18359375 disk.md0p1.out‚ 876.78125 disk.sdb2.in‚ 0.6328125 disk.sda.in‚ 3.55078125* proc.memory.percent‚17.862582206726074 disk.sda2.in‚ 3.55078125 disk.rbd2.in‚0# disk.nvme0n1.out‚85536.95703125 disk.nvme1n1.in‚ 744.32421875$ memory_percent‚91.29595756530762( proc.memory.availableMB‚ 1426.0703125 network.recv‚ 2.689668e+06 disk.dm-0.out‚122198.3984375 disk.nvme0n1.in‚ 939.5390625 disk.md0.in‚ 4.18359375 disk.rbd2.out‚2.1875# proc.memory.rssMB‚ 2926.60546875 disk.sdb.out‚911.4462890625 disk.sda2.out‚911.4462890625# disk.nvme1n1.out‚36661.44140625á_ßR2è- _wandbruntime‚8189  _runtime‚8189ì©èÎÉéà _step‚571 ' lr-AdamW/pg1‚0.00047317115160635197 ' lr-AdamW/pg2‚0.00047317115160635197 trainer/global_step‚6999 % _timestamp‚1.7885041316164281e+09 _runtime‚8189.393767653,ùcèÎêÈ ' lr-AdamW/pg2‚0.00047317115160635197 trainer/global_step‚6999 % _timestamp‚1.7885041316164281e+09 _runtime‚8189.393767653 _step‚571 ' lr-AdamW/pg1‚0.00047317115160635197»•(F2ë- _wandbruntime‚8189  _runtime‚81893˜ü'ìöO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚596455885859328 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚638 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263207936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚588449643277824 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8364637051392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8007170555904 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚577005540126048 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129839287 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 177088208 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚657689598801920 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8007170555904 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 177061864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 323093942 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87474080 _step‚572 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 146005734 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚596455885859328 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚649325201018368 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87474078 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42365207 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚649325224958464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚588449625274880 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8364397783552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 % _timestamp‚1.7885041316176503e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8006260584448 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚596456813833728 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146005734 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚6999 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚569029236596144 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146005732 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 129838647 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _runtime‚8189.395029898 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 323067596 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚577006465140412 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 177088208 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177061864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7976320569264 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7977228544268 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚596456813833728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185–Ò|x(íûO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 146005734 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚596455885859328 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚649325201018368 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87474078 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42365207 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚649325224958464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚588449625274880 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8364397783552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 % _timestamp‚1.7885041316176503e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8006260584448 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚596456813833728 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146005734 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚6999 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚569029236596144 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146005732 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 129838647 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _runtime‚8189.395029898 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 323067596 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚577006465140412 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 177088208 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177061864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7976320569264 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7977228544268 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚596456813833728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚596455885859328 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚638 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263207936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚588449643277824 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8364637051392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8007170555904 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚577005540126048 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129839287 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 177088208 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚657689598801920 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8007170555904 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 177061864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 323093942 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87474080 _step‚572¼­€ÿR‘î‚(:† ÄÐéÔóÿm gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.61' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09Ü}kï‚(:à ÄÐéÔšÙn disk.rbd0.in‚0 network.sent‚ 1.7948764e+07 network.recv‚ 2.693831e+06$ disk./.usageGB‚2711.137825012207 disk.sda.out‚912.85400390625 disk.md0p1.in‚ 4.18359375 disk.md0p1.out‚ 878.1328125 disk.dm-0.in‚ 1683.86328125 disk.rbd2.out‚2.1875 disk.sdb1.in‚0" disk.nvme0n1.out‚ 85536.9609375 disk.md0.in‚ 4.18359375 cpu‚25.170680822246506) disk./.usagePercent‚75.80454828667348 disk.sda.in‚ 3.55078125 disk.sda2.in‚ 3.55078125! disk.sda2.out‚912.85400390625# disk.nvme1n1.out‚36661.61328125' proc.memory.availableMB‚ 1426.078125! disk.dm-0.out‚122198.57421875 disk.sdb.in‚ 0.6328125 disk.nvme0n1.in‚ 939.5390625 disk.nvme1n1.in‚ 744.32421875 disk.sda1.in‚0 disk.rbd0.out‚2.1875 disk.sdb2.in‚ 0.6328125! disk.sdb2.out‚912.85400390625 disk.md0.out‚ 878.1328125# memory_percent‚91.2959098815918# proc.memory.rssMB‚ 2926.60546875 proc.cpu.threads‚17 disk.sdb.out‚912.85400390625* proc.memory.percent‚17.862582206726074 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sdb1.out‚0 &l2ð- _wandbruntime‚8190  _runtime‚8190‘Xk®ñ€N SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129856556 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚569108237151328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/ac¢QXŠXtive.all.peak‚32249 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 323138959 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 % _timestamp‚1.7885041327672038e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚649415450662912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚577085674993870 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚577086600008442 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 129857195 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 177113065 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 177113065 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚657781248085504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚596539554393088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8007382044672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 323138959 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚596538626405376 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 177086695 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚649415426722816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _runtime‚8190.545435507 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146025892 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146025894 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚588531262363648 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784§ 'ò…N SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129856556 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚569108237151328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 323138959 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 % _timestamp‚1.7885041327672038e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚649415450662912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚577085674993870 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚577086600008442 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 129857195 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 177113065 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 177113065 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚657781248085504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚596539554393088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8007382044672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 323138959 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚596538626405376 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 177086695 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚649415426722816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _runtime‚8190.545435507 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146025892 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146025894 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚588531262363648 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784½y~¿&2ó- _wandbruntime‚8190  _runtime‚8190Äô§Ç]ô× 1loss/SRqqt2/detection_loss‚2.3460769078561237 1loss/SRqqt3/detection_loss‚0.9316113331010492 2loss/SRqqt4/assignment_loss‚1.5622740826186012 .loss/FRt1/detection_loss‚1.844404210411588 /loss/FRt2/detection_loss‚2.2675670951915285 /loss/FRt3/detection_loss‚1.7735101565587938 /loss/FRt4/detection_loss‚3.5262855917215346 trainer/global_step‚6999 % _timestamp‚1.7885041327695417e+09 1loss/SRqqt4/detection_loss‚3.1603757037835964 .loss/FBt3/detection_loss‚3.579528719186783 .loss/FBt2/detection_loss‚2.132549531319562 _step‚574 /loss/FRt3/assignment_loss‚1.676017243611185 0loss/FRt4/assignment_loss‚3.7687871903181076 1loss/FBt2/assignment_loss‚0.46246983207803327 1loss/FBt3/assignment_loss‚0.23854395904345438 epoch‚5 0loss/FRt2/assignment_loss‚3.4284122784932456 /loss/FBt4/detection_loss‚2.1985089580217996 &loss/total_loss‚44.149121888338826 /loss/FBt1/detection_loss‚0.6515721173255123 2loss/SRqqt2/assignment_loss‚1.5963313399947114 _runtime‚8190.546525471 0loss/FRt1/assignment_loss‚2.3956574467902487 2loss/SRqqt1/assignment_loss‚1.1111947344916482 1loss/SRqqt3/assignment_loss‚0.908910699562705 1loss/FBt1/assignment_loss‚0.17548345560807438 0loss/FBt4/assignment_loss‚0.3744978466684188 0loss/SRqqt1/detection_loss‚2.0385514545825221؉ObõÜ 0loss/FRt1/assignment_loss‚2.3956574467902487 2loss/SRqqt1/assignment_loss‚1.1111947344916482 1loss/SRqqt3/assignment_loss‚0.908910699562705 1loss/FBt1/assignment_loss‚0.17548345560807438 0loss/FBt4/assignment_loss‚0.3744978466684188 0loss/SRqqt1/detection_loss‚2.038551454582522 1loss/SRqqt2/detection_loss‚2.3460769078561237 1loss/SRqqt3/detection_loss‚0.9316113331010492 2loss/SRqqt4/assignment_loss‚1.5622740826186012 .loss/FRt1/detection_loss‚1.844404210411588 /loss/FRt2/detection_loss‚2.2675670951915285 /loss/FRt3/detection_loss‚1.7735101565587938 /loss/FRt4/detection_loss‚3.5262855917215346 trainer/global_step‚6999 % _timestamp‚1.7885041327695417e+09 1loss/SRqqt4/detection_loss‚3.1603757037835964 .loss/FBt3/detection_loss‚3.579528719186783 .loss/FBt2/detection_loss‚2.132549531319562 _step‚574 /loss/FRt3/assignment_loss‚1.676017243611185 0loss/FRt4/assignment_loss‚3.7687871903181076 1loss/FBt2/assignment_loss‚0.46246983207803327 1loss/FBt3/assignment_loss‚0.23854395904345438 epoch‚5 0loss/FRt2/assignment_loss‚3.4284122784932456 /loss/FBt4/detection_loss‚2.1985089580217996 &loss/total_loss‚44.149121888338826 /loss/FBt1/detection_loss‚0.6515721173255123 2loss/SRqqt2/assignment_loss‚1.5963313399947114 _runtime‚8190.546525471¾É,Š³“ö‚(:ˆ ÓÐéÔœû¼l gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄxFm÷‚(:â ÓÐéÔççÆn disk.rbd0.in‚0* proc.memory.percent‚17.862582206726074 disk.sda.out‚914.80615234375 disk.sdb1.out‚0 disk.rbd2.out‚2.1875! disk.sda2.out‚914.80615234375 disk.nvme1n1.in‚ 744.32421875 disk.sdb1.in‚0 disk.md0p1.out‚ 880.01953125 disk.dm-0.out‚122677.2578125# memory_percent‚91.2975549697876% disk./.usageGB‚2711.0597076416016 disk.sdb2.in‚ 0.6328125 disk.sda1.out‚0 disk.md0p1.in‚ 4.18359375# proc.memory.rssMB‚ 2926.60546875 proc.cpu.threads‚17 network.sent‚ 1.8057692e+07 network.recv‚ 2.699013e+06 disk.dm-0.in‚ 1683.86328125 disk.sdb.in‚ 0.6328125 cpu‚25.170461783258027 disk.rbd2.in‚0" disk.nvme1n1.out‚ 36896.2109375 disk.md0.in‚ 4.18359375 disk.md0.out‚ 880.01953125 disk.sda1.in‚0) proc.memory.availableMB‚ 1425.80859375 disk.sda2.in‚ 3.55078125 disk.sda.in‚ 3.55078125 disk.rbd0.out‚2.1875 disk.sdb.out‚914.80615234375) disk./.usagePercent‚75.80236409229668 disk.nvme0n1.in‚ 939.5390625! disk.nvme0n1.out‚ 85781.046875! disk.sdb2.out‚914.80615234375†0:Ù“ø‚(:ˆ âÐéÔõ·“l gpu.0.gpu‚24.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚177.231' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.30775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A¯Ÿ-gù‚(:Ü âÐéÔ×Éál# disk.nvme0n1.out‚85903.89453125" disk.nvme1n1.out‚ 36945.7421875 disk.sda.in‚ 3.55078125 cpu‚25.17019345133227 network.sent‚ 1.8142459e+07$ disk./.usageGB‚2710.607955932617 disk.sdb2.in‚ 0.6328125 disk.sda.out‚916.2841796875 disk.md0p1.in‚ 4.18359375* proc.memory.percent‚17.862582206726074 disk.sdb.in‚ 0.6328125 disk.sda2.in‚ 3.55078125 disk.sda2.out‚916.2841796875 disk.sdb1.in‚0 disk.md0p1.out‚ 881.4453125( disk./.usagePercent‚75.7897329254407 disk.sdb.out‚916.2841796875 disk.nvme1n1.in‚ 744.32421875 disk.md0.out‚ 881.4453125 disk.sdb2.out‚916.2841796875 disk.sda1.out‚0 disk.rbd0.out‚2.1875 disk.sdb1.out‚0! disk.dm-0.out‚122849.63671875 disk.nvme0n1.in‚ 939.5390625) proc.memory.availableMB‚ 1426.01171875# proc.memory.rssMB‚ 2926.60546875 network.recv‚ 2.70494e+06 disk.dm-0.in‚ 1683.86328125 proc.cpu.threads‚17 disk.rbd2.out‚2.1875 disk.sda1.in‚0 disk.md0.in‚ 4.18359375 disk.rbd2.in‚0 disk.rbd0.in‚0$ memory_percent‚91.29631519317627¯ñ)‘“ú‚(:ˆ ñÐéÔè ël gpu.0.gpu‚20.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚156.911' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.22775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08$ø_û‚(:Ô ñÐéÔ€ÓÕm disk.sdb.in‚ 0.6328125" disk.nvme1n1.out‚ 36946.7578125 disk.sdb2.in‚ 0.6328125 disk.rbd0.out‚2.1875 disk.md0p1.in‚ 4.18359375( proc.memory.availableMB‚ 1426.1328125 disk.sda.in‚ 3.55078125 disk.sda1.in‚0 cpu‚25.169939120175293$ disk./.usageGB‚2710.557342529297 disk.sdb.out‚ 918.0859375 disk.nvme1n1.in‚ 744.32421875 disk.md0.out‚ 883.18359375 disk.rbd2.out‚2.1875 disk.sda1.out‚0* proc.memory.percent‚17.862582206726074 proc.cpu.threads‚17) disk./.usagePercent‚75.78831775350051 disk.md0p1.out‚ 883.18359375 disk.dm-0.in‚ 1683.86328125# disk.nvme0n1.out‚86148.02734375 disk.rbd2.in‚0 disk.sda2.in‚ 3.55078125 disk.nvme0n1.in‚ 939.5390625 disk.sdb2.out‚ 918.0859375! disk.dm-0.out‚123094.78515625$ memory_percent‚91.29557609558105# proc.memory.rssMB‚ 2926.60546875 disk.md0.in‚ 4.18359375 disk.sda.out‚ 918.0859375 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sda2.out‚ 918.0859375 disk.sdb1.in‚0 network.sent‚ 1.8149919e+07 network.recv‚ 2.708105e+06«SÄ2ü- _wandbruntime‚8247  _runtime‚8247…³üXÆýÀ & lr-AdamW/pg1‚0.0004656196149723648 & lr-AdamW/pg2‚0.0004656196149723648 trainer/global_step‚7049 $ _timestamp‚1.788504189320802e+09 _runtime‚8247.098186386 _step‚575™ï2°ËþÅ & lr-AdamW/pg2‚0.0004656196149723648 trainer/global_step‚7049 $ _timestamp‚1.788504189320802e+09 _runtime‚8247.098186386 _step‚575 & lr-AdamW/pg1‚0.0004656196149723648¿ßuÊŽ2ÿ- _wandbruntime‚8247  _runtime‚8247Ìo1[€ öO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147013734 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8423035473920 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 178306766 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 % _timestamp‚1.7885041893224926e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 325320498 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 325346844 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚592530597569024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚576 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚592530597569024 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚592530579566080 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 130733418 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚653836510180864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8062348009472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8063257980928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚662259761853952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚8247.100169704 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178333110 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42655471 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚600592927575552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178333110 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8423251673088 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8033105799376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 130733418 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚653836510180864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8062348009472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8063257980928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚662259761853952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚8247.100169704 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178333110 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42655471 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚600592927575552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178333110 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8423251673088 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8033105799376 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚592530597569024ÀoP•Q2‚ - _wandbruntime‚8248  _runtime‚8248’5AN'ƒ ÿM PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚600676596109312 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147033894 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 325365489 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚581092431976178 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 325391861 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8063469469696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130750671 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88089393 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚600676596109312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178331597 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 % _timestamp‚1.7885041904854426e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚592612216654848 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8064379454464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 325391861 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130751316 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚573059133917888 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚573059116878528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8063469469696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147033892 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚653926735885312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚592612216654848 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147033892 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚592612198651904 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 178357967 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚600676596109312 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147033894 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 325365489 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚581092431976178 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 325391861 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8063469469696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130750671 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88089393 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚600676596109312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178331597 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 % _timestamp‚1.7885041904854426e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚592612216654848 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8064379454464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 325391861 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130751316 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚573059133917888 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚573059116878528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8063469469696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147033892 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚653926735885312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚592612216654848 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147033892 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚592612198651904 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 178357967 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚666829441327616 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚604729969291776 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 327573400 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚596611533857280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8119345405952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240139264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚621 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8119345405952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 148021734 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148021732 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8481886063616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚596611551860224 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179551668 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚585019979105028 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚576930979011184 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚658347771463168 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚585019054090664 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚666829681466880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 % _timestamp‚1.7885042467900517e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 131626759 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚604729969291776 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 148021732 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8481669864448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216199168 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚576930996050544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88680580 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚658347795403264 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0uXÇ-(• ûO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179551668 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8118435434496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 179578012 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 179578012 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 131627380 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42946181 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8118435434496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚666829441327616 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚604729969291776 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 327573400 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚596611533857280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8119345405952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240139264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚621 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8119345405952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 148021734 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148021732 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8481886063616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚596611551860224 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179551668 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚585019979105028 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚576930979011184 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚658347771463168 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚585019054090664 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚666829681466880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 % _timestamp‚1.7885042467900517e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 131626759 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚604729969291776 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 148021732 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8481669864448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216199168 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚576930996050544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88680580 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚658347795403264 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0ÄÅF12– - _wandbruntime‚8305  _runtime‚8305¯i— ÿM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚658438021107712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚596693152943104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88692645 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚604812709837824 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 148041892 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148041894 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42951962 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚585100113973058 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 H@DeÈÙ]Vœ viceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 327644763 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚577010013645088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚577009996605728 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 327618391 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8483072979968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42952599 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8119556894720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88692643 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _runtime‚8305.672335564 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148041894 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 179576499 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚596693152943104 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚666920843475968 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148041892 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚585099188958486 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚604813637825536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8089192352758 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚596693170946048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8482846308352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8120466879488 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8090100327970 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _step‚581 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179602869 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 131645244 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8119556894720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 trainer/global_step‚7099 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 327618391 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131644605 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 327644763 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 $ _timestamp‚1.788504247894343e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179602869 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚666921094087680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131644605 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 327644763 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 $ _timestamp‚1.788504247894343e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179602869 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚666921094087680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 327644763 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚577010013645088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚577009996605728 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 327618391 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8483072979968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42952599 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8119556894720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88692643 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _runtime‚8305.672335564 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148041894 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 179576499 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚596693152943104 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚666920843475968 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148041892 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚585099188958486 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚604813637825536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8089192352758 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚596693170946048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8482846308352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8120466879488 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8090100327970 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _step‚581 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179602869 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 131645244 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8119556894720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0Å #ËŠ2™ - _wandbruntime‚8305  _runtime‚8305bÍ\Wš Ñ 1loss/SRqqt4/assignment_loss‚2.120133084555467 0loss/FRt3/assignment_loss‚1.5121453618391967 0loss/FBt2/assignment_loss‚0.2663987632179972 ,loss/FBt1/detection_loss‚0.7439590802632 %loss/total_loss‚41.39789101404652 _step‚582 1loss/FBt1/assignment_loss‚0.11185885722245087 /loss/FRt2/detection_loss‚1.9379198736009688 /loss/FRt4/detection_loss‚2.8802233073446484 1loss/SRqqt1/detection_loss‚2.2872454039752483 1loss/SRqqt3/detection_loss‚1.0200417029586706 /loss/FRt1/detection_loss‚1.8961339847592171 .loss/FRt1/assignment_loss‚2.09226373151729 /loss/FRt3/detection_loss‚1.6686521410846558 trainer/global_step‚7099 _runtime‚8305.675154237 2loss/SRqqt2/assignment_loss‚1.4711433025076985 1loss/SRqqt4/detection_loss‚2.5679970262227236 epoch‚5 $ _timestamp‚1.788504247897661e+09 0loss/FBt4/assignment_loss‚0.3271862603560294 0loss/FRt2/assignment_loss‚3.0707376496149945 /loss/FRt4/assignment_loss‚3.440204633606805 /loss/FBt2/detection_loss‚2.5399304976065955 /loss/FBt4/detection_loss‚1.9608741228779156 2loss/SRqqt1/assignment_loss‚1.3816388668492436 2loss/SRqqt3/assignment_loss‚1.0125418099871075 1loss/FBt3/assignment_loss‚0.26097467681808795 0loss/SRqqt2/detection_loss‚2.559658071026206 .loss/FBt3/detection_loss‚2.268028804234096箼 \› Ö 1loss/FBt1/assignment_loss‚0.11185885722245087 /loss/FRt2/detection_loss‚1.9379198736009688 /loss/FRt4/detection_loss‚2.8802233073446484 1loss/SRqqt1/detection_loss‚2.2872454039752483 1loss/SRqqt3/detection_loss‚1.0200417029586706 /loss/FRt1/detection_loss‚1.8961339847592171 .loss/FRt1/assignment_loss‚2.09226373151729 /loss/FRt3/detection_loss‚1.6686521410846558 trainer/global_step‚7099 _runtime‚8305.675154237 2loss/SRqqt2/assignment_loss‚1.4711433025076985 1loss/SRqqt4/detection_loss‚2.5679970262227236 epoch‚5 $ _timestamp‚1.788504247897661e+09 0loss/FBt4/assignment_loss‚0.3271862603560294 0loss/FRt2/assignment_loss‚3.0707376496149945 /loss/FRt4/assignment_loss‚3.440204633606805 /loss/FBt2/detection_loss‚2.5399304976065955 /loss/FBt4/detection_loss‚1.9608741228779156 2loss/SRqqt1/assignment_loss‚1.3816388668492436 2loss/SRqqt3/assignment_loss‚1.0125418099871075 1loss/FBt3/assignment_loss‚0.26097467681808795 0loss/SRqqt2/detection_loss‚2.559658071026206 .loss/FBt3/detection_loss‚2.268028804234096 1loss/SRqqt4/assignment_loss‚2.120133084555467 0loss/FRt3/assignment_loss‚1.5121453618391967 0loss/FBt2/assignment_loss‚0.2663987632179972 ,loss/FBt1/detection_loss‚0.7439590802632 %loss/total_loss‚41.39789101404652 _step‚582Æ%KÀkœ ‚(:à ¼ÑéÔ¥úËm% disk./.usageGB‚2711.1383514404297 disk.dm-0.out‚124149.0859375 network.sent‚ 1.8279189e+07 disk.md0p1.in‚ 4.18359375 disk.nvme1n1.in‚ 744.32421875 disk.md0.out‚ 891.51953125 disk.rbd2.out‚2.265625* proc.memory.percent‚17.862582206726074 disk.sdb2.in‚ 0.6328125 disk.sda2.in‚ 3.55078125 disk.sda2.out‚926.7705078125' proc.memory.availableMB‚ 1425.890625 disk.nvme0n1.in‚ 939.5390625 disk.sdb1.out‚0 network.recv‚ 2.730836e+06 disk.sda.in‚ 3.55078125# proc.memory.rssMB‚ 2926.60546875 disk.sda.out‚926.7705078125 disk.nvme1n1.out‚ 36948.21875$ memory_percent‚91.29705429077148 disk.sdb2.out‚926.7705078125 disk.sdb1.in‚0 disk.sdb.out‚926.7705078125 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.md0p1.out‚ 891.51953125 disk.md0.in‚ 4.18359375 cpu‚25.16868044003956" disk.nvme0n1.out‚ 87200.8671875 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 2.26171875 disk.dm-0.in‚ 1683.86328125 disk.sdb.in‚ 0.6328125( disk./.usagePercent‚75.8045630058269±Á%ב ‚(:† ¼ÑéÔÉÞ’n gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚180.196' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.049 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AÁÉ“ž ‚(:ˆ ËÑéÔÆþÑm gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚178.629' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.65725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sC³KgŸ ‚(:Ü ËÑéÔ¶¦ím disk.sdb.out‚ 928.2734375 disk.rbd2.out‚ 2.35302734375 disk.sda1.in‚0$ memory_percent‚91.29681587219238 network.sent‚ 1.8385716e+07 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.md0p1.out‚ 892.953125( disk./.usagePercent‚75.8045630058269 disk.md0.in‚ 4.18359375 disk.sdb2.in‚ 0.6328125 disk.sdb2.out‚ 928.2734375 disk.sda.out‚ 928.2734375 proc.cpu.threads‚17 disk.dm-0.out‚ 124524.078125 disk.sda2.out‚ 928.2734375 disk.nvme1n1.in‚ 744.32421875 disk.rbd2.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 1425.9296875# proc.memory.rssMB‚ 2926.60546875 network.recv‚ 2.735291e+06% disk./.usageGB‚2711.1383514404297# disk.nvme0n1.out‚87575.15234375# disk.nvme1n1.out‚36948.92578125 disk.md0p1.in‚ 4.18359375 disk.md0.out‚ 892.953125 disk.dm-0.in‚ 1683.86328125 disk.sdb.in‚ 0.6328125 cpu‚25.168405774495582 disk.sdb1.out‚0 disk.sda2.in‚ 3.55078125 disk.nvme0n1.in‚ 939.5390625 disk.sda.in‚ 3.55078125* proc.memory.percent‚17.862582206726074 disk.rbd0.out‚ 2.3408203125¨Í“  ‚(:ˆ ÚÑéÔÄÔm gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zT7Œy¡ ‚(:î ÚÑéÔ„ïšm disk.rbd2.out‚ 2.35302734375! disk.sdb2.out‚929.61669921875# disk.nvme0n1.out‚87668.22265625# memory_percent‚91.2982702255249) proc.memory.availableMB‚ 1425.69140625 disk.sdb2.in‚ 0.6328125 disk.sda.in‚ 3.55078125 disk.rbd0.out‚ 2.3408203125 disk.sdb1.in‚0 disk.md0p1.out‚ 894.2421875! disk.nvme1n1.out‚ 36949.078125 cpu‚25.168167626066754 proc.cpu.threads‚17 network.sent‚ 1.8394193e+07 disk.sda.out‚929.61669921875 disk.sda2.in‚ 3.55078125 disk.md0.out‚ 894.2421875* proc.memory.percent‚17.862582206726074% disk./.usageGB‚2711.1383514404297 disk.sda1.out‚0! disk.sda2.out‚929.61669921875 disk.md0.in‚ 4.18359375# proc.memory.rssMB‚ 2926.60546875( disk./.usagePercent‚75.8045630058269 disk.rbd2.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 1683.86328125 disk.nvme1n1.in‚ 744.32421875 disk.sdb.in‚ 0.6328125 disk.sdb.out‚929.61669921875 disk.nvme0n1.in‚ 939.5390625 network.recv‚ 2.740077e+06 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚ 4.18359375! disk.dm-0.out‚124617.30078125æÅüj‘¢ ‚(:† éÑéÔÌûÌm gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.548' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.637 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0" t£ ‚(:é éÑéÔÎÚ¿n disk.rbd0.in‚0 disk.sda2.out‚931.5087890625 disk.nvme0n1.in‚ 939.5390625" disk.nvme1n1.out‚ 36949.2421875 disk.rbd2.in‚0 disk.rbd2.out‚ 2.41552734375 disk.sdb1.in‚0) disk./.usagePercent‚75.78975468418926 disk.dm-0.in‚ 1683.86328125 disk.sda2.in‚ 3.55078125( proc.memory.availableMB‚ 1425.9296875* proc.memory.percent‚17.862582206726074% disk./.usageGB‚2710.6087341308594 disk.sdb2.in‚ 0.6328125 disk.sdb2.out‚931.5087890625 disk.sda1.in‚0 disk.dm-0.out‚ 125044.453125 disk.sda.out‚931.5087890625 disk.md0p1.out‚ 896.0703125 disk.nvme1n1.in‚ 744.32421875 disk.sdb1.out‚0 network.recv‚ 2.743555e+06 disk.rbd0.out‚ 2.40380859375" disk.nvme0n1.out‚ 88095.2109375$ memory_percent‚91.29681587219238# proc.memory.rssMB‚ 2926.60546875 cpu‚25.167948049734417 proc.cpu.threads‚17 disk.sda.in‚ 3.55078125 disk.sda1.out‚0 disk.sdb.out‚931.5087890625 disk.md0.in‚ 4.18359375 disk.md0.out‚ 896.0703125 network.sent‚ 1.839951e+07 disk.md0p1.in‚ 4.18359375 disk.sdb.in‚ 0.6328125$qY2¤ - _wandbruntime‚8362  _runtime‚8362Ž7¼mÉ¥ à _runtime‚8362.243654964 _step‚583 ' lr-AdamW/pg1‚0.00045055597292541135 ' lr-AdamW/pg2‚0.00045055597292541135 trainer/global_step‚7149 % _timestamp‚1.7885043044663198e+09_ÜþΦ È ' lr-AdamW/pg1‚0.00045055597292541135 ' lr-AdamW/pg2‚0.00045055597292541135 trainer/global_step‚7149 % _timestamp‚1.7885043044663198e+09 _runtime‚8362.243654964 _step‚583Ç`Sè2§ - _wandbruntime‚8362  _runtime‚83629#mõ¨ öO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚662859056685568 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 329826302 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89283830 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚608867011008000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89283828 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚600692506151424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149029734 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8144860309592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8175432830976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚608867938982400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8174522859520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚600692506151424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 180822914 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8540113559552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚584 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚600692488148480 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43234720 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚671399170245120 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚580881858738384 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570‰E”É(© ûO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 180822914 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚580881858738384 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚662859056685568 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 329826302 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89283830 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚608867011008000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89283828 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚600692506151424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149029734 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8144860309592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8175432830976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚608867938982400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8174522859520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚600692506151424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 180822914 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8540113559552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚584 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚600692488148480 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43234720 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚671399170245120 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8176554304512 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚600774107234304 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329897665 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚580960893372288 _runtime‚8363.294193478 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 149049894 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 132536984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚600774107234304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚671490819826688 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43240480 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132536373 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43241089 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8145069607866 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚608949751554048 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚662949282390016 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8541280533504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8145977583078 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 180847771 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚600774125237248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 329871293 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149049892 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 180821401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚611 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788504305514911e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 180821401 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8541513496576 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89295895 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8176554304512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚662949282390016 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8541280533504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8145977583078 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 180847771 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚600774125237248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 329871293 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149049892 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 180821401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚611 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788504305514911e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 180821401 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8541513496576 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89295895 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8176554304512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8176554304512 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚600774107234304 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_end/d„‘l reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329897665 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚580960893372288 _runtime‚8363.294193478 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 149049894 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 132536984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚600774107234304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚671490819826688 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43240480 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132536373 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43241089 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8145069607866 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚608949751554048 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150037734 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚675969296514048 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚588 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 133412349 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89887080 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚667370341907968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8598930665984 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8598695592448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89887078 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚613004052724224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚604773442439680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259013632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚604773460442624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚667370365848064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 182067816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 _runtime‚8419.650983314 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 332105550 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 $ _timestamp‚1.788504361869708e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚613004980698624 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150037732 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 182041472 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8230610284544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268Îc:)'½ úO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚613004052724224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚604773442439680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259013632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚604773460442624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚667370365848064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 182067816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 _runtime‚8419.650983314 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 332105550 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 $ _timestamp‚1.788504361869708e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚613004980698624 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150037732 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 182041472 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8230610284544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 trainer/global_step‚7199 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 182067816 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚675969037500416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8200737564700 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚604773442439680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚613004980698624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8231520256000 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚584832738465584 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150037734 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚675969296514048 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚588 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 133412349 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89887080 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚667370341907968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8598930665984 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8598695592448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/inactive_sp9E·Ð×lit.large_pool.freed‚89887078 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640ÌÊ3+—2¾ - _wandbruntime‚8420  _runtime‚8420W7‹¨'¿ ÿM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 182092673 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89899143 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133430225 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 _runtime‚8420.610732981 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8600098832896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8599882647040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8231731744768 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150057892 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚604855079528448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8231731744768 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332150567 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _step‚589 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 182092673 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚613086793270272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚622 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚667460567612416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚593112702923102 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 332150567 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 332124195 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150057894 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 182066303 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43531082 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43531704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚604855061525504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚584911773099488 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚676060690385408 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚593113627937674 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚604855061525504 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚676060450259456 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚613087721257984 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚667460591552512 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0/}\4 'À „N =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8201854838186 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8200946862974 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 $ _timestamp‚1.788504362832764e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚604855061525504 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚676060450259456 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚613087721257984 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚667460591552512 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 182092673 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89899143 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133430225 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 _runtime‚8420.610732981 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8600098832896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8599882647040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8231731744768 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150057892 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚604855079528448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8231731744768 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332150567 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _step‚589 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 182092673 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚613086793270272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚622 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚667460567612416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚593112702923102 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 332150567 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 332124195 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150057894 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 182066303 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43531082 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43531704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚604855061525504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚584911773099488 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚676060690385408 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚593113627937674 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚680539190130688 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151045734 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151045734 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43815827 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚671881651070464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8287607681024 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 134306797 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚608854414733824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90490330 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚597040250051952 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peJ^s‰ak‚ 13684416521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8286697709568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚617141094440448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚608854414733824 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚671881627130368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 334332106 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151045732 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚617142022414848 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183312718 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚588783635232144 % _timestamp‚1.7885044194618967e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 183312718 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8286697709568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚680538941602816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 334358452 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚608854396730880 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43816467 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8657539060224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚588783618192784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151045732 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 134306155 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 trainer/global_step‚7249 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚608854396730880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8287607681024®œ (Ñ ûO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _runtime‚8477.239647465 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚680539190130688 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151045734 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151045734 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43815827 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚671881651070464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8287607681024 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 134306797 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚608854414733824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90490330 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚597040250051952 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8286697709568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚617141094440448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚608854414733824 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚671881627130368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 334332106 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151045732 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚617142022414848 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183312718 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚588783635232144 % _timestamp‚1.7885044194618967e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 183312718 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8286697709568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚680538941602816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 334358452 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚608854396730880 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43816467 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8657539060224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚588783618192784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151045732 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 134306155 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 trainer/global_step‚7249 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚608854396730880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8287607681024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _step‚592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 183286374Ð7 à2Ò - _wandbruntime‚8478  _runtime‚8478æ×~'Ó €N QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 _runtime‚8478.395178538 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 183311205 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚671971876774912 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚608936015816704 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151065892 % _timestamp‚1.7885044206159546e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134324047 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 334377097 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43822303 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _step‚593 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚608936033819648 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90502395 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚617223834986496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334403469 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90502393 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8288729154560 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚608936033819648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151065894 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 183337575 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8287819169792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8658461201920 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 trainer/global_step‚7249 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚597120384919982 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚617224762974208 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183311205 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚680630314036736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚680630577231360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8256824118082 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183337575 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8287819169792 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151065894 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _step‚593 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚608936033819648 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90502395 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚617223834986496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334403469 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90502393 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8288729154560 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚608936033819648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151065894 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 183337575 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8287819169792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8658461201920 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 MCDeviceStatsMonitor.on_train_batch_end/allocatioôMðn.small_pool.current‚26370 trainer/global_step‚7249 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚597120384919982 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚617224762974208 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183311205 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚680630314036736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚680630577231360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8256824118082 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183337575 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8287819169792 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151065894 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 _runtime‚8478.395178538 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 183311205 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚671971876774912 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚608936015816704 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151065892 % _timestamp‚1.7885044206159546e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134324047ÑÌÁcú2Õ - _wandbruntime‚8478  _runtime‚8478 w.:YÖ Ó 2loss/SRqqt1/assignment_loss‚1.5846915344397228 1loss/SRqqt2/detection_loss‚2.5529855095914433 _step‚594 2loss/SRqqt3/assignment_loss‚0.9282403968609287 0loss/FBt3/assignment_loss‚0.2113911532131689 trainer/global_step‚7249 % _timestamp‚1.7885044206192193e+09 0loss/FRt1/assignment_loss‚2.4006732719786026 /loss/FRt2/assignment_loss‚2.908279319019879 .loss/FRt2/detection_loss‚2.667151330586742 1loss/SRqqt3/detection_loss‚0.9245648987707682 0loss/FBt1/assignment_loss‚0.2627138041425263 /loss/FRt3/detection_loss‚1.5990466359948767 /loss/FBt3/detection_loss‚2.3252578462873186 epoch‚5 0loss/FBt4/assignment_loss‚0.1752720908047865 1loss/SRqqt4/detection_loss‚2.5666399598121643 /loss/FBt1/detection_loss‚0.7002124967149053 .loss/FBt4/detection_loss‚2.499158467117109 $loss/total_loss‚41.8863700475833 _runtime‚8478.396501873 /loss/FRt4/assignment_loss‚2.810296053236181 2loss/SRqqt2/assignment_loss‚1.8006779117775815 1loss/FBt2/assignment_loss‚0.23400640234004558 .loss/FRt1/detection_loss‚2.090030601782286 /loss/FRt4/detection_loss‚3.1960411586544732 /loss/FRt3/assignment_loss‚1.203331450621287 2loss/SRqqt4/assignment_loss‚1.2544621920585632 1loss/SRqqt1/detection_loss‚2.3018594053056507 /loss/FBt2/detection_loss‚2.6893861564722927IÎ^× Ø 0loss/FBt4/assignment_loss‚0.1752720908047865 1loss/SRqqt4/detection_loss‚2.5666399598121643 /loss/FBt1/detection_loss‚0.7002124967149053 .loss/FBt4/detection_loss‚2.499158467117109 $loss/total_loss‚41.8863700475833 _runtime‚8478.396501873 /loss/FRt4/assignment_loss‚2.810296053236181 2loss/SRqqt2/assignment_loss‚1.8006779117775815 1loss/FBt2/assignment_loss‚0.23400640234004558 .loss/FRt1/detection_loss‚2.090030601782286 /loss/FRt4/detection_loss‚3.1960411586544732 /loss/FRt3/assignment_loss‚1.203331450621287 2loss/SRqqt4/assignment_loss‚1.2544621920585632 1loss/SRqqt1/detection_loss‚2.3018594053056507 /loss/FBt2/detection_loss‚2.6893861564722927 2loss/SRqqt1/assignment_loss‚1.5846915344397228 1loss/SRqqt2/detection_loss‚2.5529855095914433 _step‚594 2loss/SRqqt3/assignment_loss‚0.9282403968609287 0loss/FBt3/assignment_loss‚0.2113911532131689 trainer/global_step‚7249 % _timestamp‚1.7885044206192193e+09 0loss/FRt1/assignment_loss‚2.4006732719786026 /loss/FRt2/assignment_loss‚2.908279319019879 .loss/FRt2/detection_loss‚2.667151330586742 1loss/SRqqt3/detection_loss‚0.9245648987707682 0loss/FBt1/assignment_loss‚0.2627138041425263 /loss/FRt3/detection_loss‚1.5990466359948767 /loss/FBt3/detection_loss‚2.3252578462873186 epoch‚5ÒÂEg’Ø ‚(:‡ ðÒéÔ利n gpu.0.gpu‚52.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.9915 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^ú¹ dÙ ‚(:Ù ðÒéÔ½Ò–o) disk./.usagePercent‚75.78973420536708 disk.rbd0.out‚ 2.466796875 disk.dm-0.in‚ 1683.86328125 disk.nvme1n1.in‚ 744.32421875 disk.rbd2.out‚ 2.47900390625) proc.memory.availableMB‚ 1347.75390625 cpu‚25.16942079974675 network.recv‚ 2.78365e+06 disk.md0p1.in‚ 4.18359375 disk.sdb.out‚ 946.359375 disk.nvme0n1.in‚ 939.5390625 disk.sdb2.in‚ 0.6328125 disk.sdb2.out‚ 946.359375 disk.sda.in‚ 3.55078125 disk.sda1.in‚0 disk.sdb1.out‚0 disk.rbd2.in‚0$ memory_percent‚91.77396297454834! proc.memory.rssMB‚ 3004.453125 disk.dm-0.out‚ 126766.328125 disk.md0.out‚ 910.33984375% disk./.usageGB‚2710.6080017089844 disk.md0p1.out‚ 910.33984375 disk.sda2.in‚ 3.55078125# disk.nvme1n1.out‚36951.29296875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda2.out‚ 946.359375 disk.sda.out‚ 946.359375 network.sent‚ 1.8653741e+07 disk.sdb.in‚ 0.6328125* proc.memory.percent‚18.337726593017578 disk.sdb1.in‚0# disk.nvme0n1.out‚89815.03515625 disk.md0.in‚ 4.18359375 proc.cpu.threads‚17#-$(œÚ ‚(:‘ ÿÒéÔñéŠm gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GÑÏlÛ ‚(:á ÿÒéÔý„âm disk.sda2.in‚ 3.55078125! disk.sda2.out‚948.33544921875 disk.nvme1n1.in‚ 744.32421875! proc.memory.rssMB‚ 3004.453125 disk.md0p1.out‚912.25 disk.dm-0.out‚ 127162.265625 disk.nvme0n1.in‚ 939.5390625 disk.rbd0.out‚ 2.466796875 disk.sdb1.in‚0 network.sent‚ 1.876334e+07% disk./.usageGB‚2710.6339149475098 disk.rbd2.out‚ 2.47900390625 disk.rbd0.in‚0 disk.md0p1.in‚ 4.18359375# disk.nvme0n1.out‚90130.16015625 disk.md0.in‚ 4.18359375 proc.cpu.threads‚17 disk.sda.in‚ 3.55078125 disk.sda.out‚948.33544921875 disk.sdb.out‚948.33544921875 disk.md0.out‚912.25! disk.sdb2.out‚948.33544921875 disk.sdb1.out‚0 disk.sdb.in‚ 0.6328125# disk.nvme1n1.out‚37032.10546875$ memory_percent‚91.77377223968506* proc.memory.percent‚18.337726593017578 cpu‚25.176282567305254 disk.sda1.out‚0 network.recv‚ 2.788494e+06( disk./.usagePercent‚75.7904587503618 disk.sda1.in‚0 disk.dm-0.in‚ 1683.86328125 disk.rbd2.in‚0) proc.memory.availableMB‚ 1347.78515625 disk.sdb2.in‚ 0.6328125ÀAT“Ü ‚(:ˆ ŽÓéÔú”êl gpu.0.gpu‚60.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×kM‰kÝ ‚(:à ŽÓéÔ⹶m" disk.nvme0n1.out‚ 90339.8046875% disk./.usageGB‚2710.6363067626953! disk.sdb2.out‚949.87353515625 disk.nvme0n1.in‚ 939.5390625 network.recv‚ 2.793599e+06 disk.sda.out‚949.87353515625 disk.rbd2.in‚0 disk.rbd2.out‚ 2.47900390625* proc.memory.percent‚18.337726593017578) disk./.usagePercent‚75.79052562651543 disk.md0p1.out‚ 913.71875" disk.nvme1n1.out‚ 37034.6953125$ memory_percent‚91.77370071411133 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda2.in‚ 3.55078125! proc.memory.rssMB‚ 3004.453125 cpu‚25.183786495859803 disk.sdb2.in‚ 0.6328125 disk.dm-0.out‚127374.5 disk.sdb.in‚ 0.6328125' proc.memory.availableMB‚ 1347.796875 disk.md0p1.in‚ 4.18359375 disk.sdb.out‚949.87353515625 disk.nvme1n1.in‚ 744.32421875 disk.md0.in‚ 4.18359375 proc.cpu.threads‚17 network.sent‚ 1.8769728e+07 disk.sda.in‚ 3.55078125 disk.rbd0.in‚0 disk.dm-0.in‚ 1683.86328125! disk.sda2.out‚949.87353515625 disk.md0.out‚ 913.71875 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚ 2.466796875×D®Ë’Þ ‚(:‡ ÓéÔ‰¨ƒl gpu.0.gpu‚23.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.9915 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚ šyß ‚(:î ÓéÔâšm proc.cpu.threads‚17) disk./.usagePercent‚75.79403049160095 disk.rbd2.in‚0 disk.sda1.out‚0 disk.md0p1.in‚ 4.18359375 network.sent‚ 1.8775026e+07 disk.nvme0n1.in‚ 939.5390625 disk.sda.out‚951.36181640625 disk.rbd0.in‚0! proc.memory.rssMB‚ 3004.453125% disk./.usageGB‚2710.7616577148438 disk.rbd2.out‚ 2.47900390625! disk.sdb2.out‚951.36181640625 disk.sda.in‚ 3.55078125 disk.md0.out‚ 915.1484375# disk.nvme0n1.out‚90340.03515625 disk.nvme1n1.in‚ 744.32421875" disk.nvme1n1.out‚ 37064.3828125) proc.memory.availableMB‚ 1347.66796875* proc.memory.percent‚18.337726593017578! disk.sda2.out‚951.36181640625 disk.sdb2.in‚ 0.6328125$ memory_percent‚91.77448749542236 network.recv‚ 2.797062e+06 disk.sdb.out‚951.36181640625 disk.sda2.in‚ 3.55078125 disk.sda1.in‚0 disk.rbd0.out‚ 2.466796875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 1683.86328125 disk.md0p1.out‚ 915.1484375 disk.sdb.in‚ 0.6328125 disk.md0.in‚ 4.18359375! disk.dm-0.out‚127404.41796875 cpu‚25.190884272560066a¬`¦2à - _wandbruntime‚8542  _runtime‚8542w¾^c*á ÝT KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1185 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3725 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 14135738368 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 135121854 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 8687442571264 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 8316547802624 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 8687606781440 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1185 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚620237824406016 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 335751836 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚91019191 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚675089865788416 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 151904858 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 151904856 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚44102661 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 183867691 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 135121635 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 698106808 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 8317230841856 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚599935740529638 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 8286138170758 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 8285457103310 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 8317230841856 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3725 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚615 % _timestamp‚1.7885044849810889e+09 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 335751836 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚620237824406016 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚219 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1268 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 683039232 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚91019193 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14401142784 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1195376640 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 164210176 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 8316547802624 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 335772549 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚44102444 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1187728393 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚611921276603392 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 183846980 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚620238525448192 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚683777308359680 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14401142784 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚615 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 151904858 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1185 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚570 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 151904856 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚675089889728512 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13684416521 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 13002869760 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 683039232 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚570 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 701042176 _runtime‚8542.759162469 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 188150272 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚611921276603392 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 409073152 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 trainer/global_step‚7253 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚683777496509952 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚591649602358880 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 183867691 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 681067448 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13205766144Ä÷Q“"â âT dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 8687442571264 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 8316547802624 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 8687606781440 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1185 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚620237824406016 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 335751836 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚91019191 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚675089865788416 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 151904858 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 151904856 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚44102661 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 183867691 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 135121635 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 698106808 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 8317230841856 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚599935740529638 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 8286138170758 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 8285457103310 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 8317230841856 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3725 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚615 % _timestamp‚1.7885044849810889e+09 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 335751836 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚620237824406016 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚219 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1268 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 683039232 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚91019193 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14401142784 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1195376640 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 164210176 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 8316547802624 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 335772549 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚44102444 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1187728393 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚611921276603392 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 183846980 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚620238525448192 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚683777308359680 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14401142784 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚615 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 151904858 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1185 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚570 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 151904856 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚675089889728512 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13684416521 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 13002869760 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 683039232 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚570 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 701042176 _runtime‚8542.759162469 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 188150272 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚611921276603392 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 409073152 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 trainer/global_step‚7253 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚683777496509952 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚591649602358880 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 183867691 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 681067448 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13205766144 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1185 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3725 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 14135738368 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 135121854 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0ÓÅЙþ2ã - _wandbruntime‚8543  _runtime‚8543^2iœm)ä çR D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚599985956738898 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3725 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698106808 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 8285945314194 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14401142784 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚683834065764352 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14401142784 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚221 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 683039232 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 151920332 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚91028755 _runtime‚8543.847367537 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚44 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚620292575163904 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 151920334 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚44107991 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 8317720692736 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚615 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚611975555513344 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 8688096632320 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1185 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 183855794 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 701042176 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 8317037653504 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 183876505 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 409073152 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 681067448 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 151920332 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 701042176 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚611975555513344 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚611975537510400 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚620293276206080 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 151920334 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 183855794 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚219 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 164210176 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13684416521 trainer/global_step‚7253 % _timestamp‚1.7885044860697432e+09 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 335796839 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 135136525 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1268 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚44107772 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 8687932422144 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚570 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14401142784 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 183876505 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚599986654845706 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚620293276206080 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚91028753 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚591700028464064 ™ë`r)å ìR JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 8317720692736 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚615 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚611975555513344 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 8688096632320 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1185 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 183855794 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 701042176 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 8317037653504 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 183876505 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 409073152 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 681067448 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 151920332 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 701042176 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚611975555513344 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚611975537510400 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚620293276206080 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 151920334 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 183855794 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚219 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 164210176 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13684416521 trainer/global_step‚7253 % _timestamp‚1.7885044860697432e+09 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 335796839 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 135136525 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1268 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚44107772 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 8687932422144 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚570 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14401142784 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 183876505 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚599986654845706 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚620293276206080 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚91028753 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚591700028464064 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚599985956738898 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3725 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698106808 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 8285945314194 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14401142784 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚683834065764352 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14401142784 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚221 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 683039232 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 151920332 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚91028755 _runtime‚8543.847367537 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚44 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚620292575163904 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 151920334 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚44107991ÔÃK,à’æ ‚(:‡ ¬ÓéÔû†™m gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚35.0 gpu.0.powerWatts‚123.095' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.77375 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žÏgç ‚(:Ü ¬ÓéÔ–ÂÛn disk.sda.in‚ 3.55078125$ memory_percent‚91.77718162536621 disk.md0.out‚ 917.01171875 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.md0p1.out‚ 917.01171875 disk.sdb2.in‚ 0.6328125 disk.nvme0n1.in‚ 939.5390625# disk.nvme1n1.out‚37204.48828125( proc.memory.availableMB‚ 1347.2265625* proc.memory.percent‚18.337726593017578 cpu‚25.19806215212395 network.sent‚ 1.8791668e+07 disk.sda1.in‚0 disk.rbd0.out‚ 2.466796875! proc.memory.rssMB‚ 3004.453125 disk.nvme1n1.in‚ 744.32421875 disk.sdb2.out‚ 953.2890625 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.rbd2.out‚ 2.47900390625( disk./.usagePercent‚75.7975414363368 disk.sdb1.in‚0 disk.sdb.in‚ 0.6328125 disk.sdb.out‚ 953.2890625 disk.sda2.in‚ 3.55078125# disk.nvme0n1.out‚90549.68359375 disk.sda.out‚ 953.2890625 disk.sda1.out‚0% disk./.usageGB‚2710.8872261047363 disk.md0p1.in‚ 4.18359375 disk.dm-0.in‚ 1683.86328125 disk.dm-0.out‚ 127754.171875 disk.sda2.out‚ 953.2890625 disk.md0.in‚ 4.18359375 network.recv‚ 2.801434e+06P1æ$2è - _wandbruntime‚8559  _runtime‚8559€QoÇFé ¤¦ <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.49269577860832214 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960132241249084 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1963990330696106 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.15909092128276825 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 _step‚597 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22762346267700195 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_2_of_7‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41484448313713074 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16926893591880798 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993895292282104 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01662825606763363 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_5‚0.11773554980754852 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.017789427191019058 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.37995636463165283 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.11243592947721481 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.13245613873004913 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3872646391391754 *jet/accuracy_1_of_1‚0.4797857701778412 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5417895317077637 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16030772030353546 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9903751611709595 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4015151858329773 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6400061845779419 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19857648015022278 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1964285671710968 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0694444477558136 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.02923472598195076 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_6‚0.9807692170143127 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.38224396109580994 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.46666663885116577 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9926935434341431 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.4041666090488434 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4401519000530243 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5455459952354431 6 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.533991277217865 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1531531661748886 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.06378600746393204 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4907407760620117 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00877861212939024 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26229479908943176 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3982967734336853 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5350038409233093 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5096095204353333 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04459172487258911 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6929824948310852 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5327968597412109 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3472222089767456 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3860090374946594 +particle/specificity‚0.9502973556518555 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9771632552146912 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322916865348816 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14784832298755646 6 Purity/1FRt2SRqqÏ6ÿÔåwt1FBt/FBt_purity‚0.982758641242981 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3965167701244354 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3847711384296417 particle/accuracy_4_of_4‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 jet/accuracy_6_of_7‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8116438388824463 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9166666269302368 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 jet/accuracy_7_of_7‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.545976996421814 particle/accuracy_5_of_7‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1629781424999237 jet/accuracy_6_of_6‚0 particle/accuracy_5_of_6‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.030860714614391327 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5543498396873474 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9750000238418579 ;"Purity/3FRt*SRqqt1FBt/event_purity‚0.008333333767950535 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.593303918838501 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5006116032600403 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_2‚0.7531417012214661 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 *"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,jet/accuracy_4_of_6‚0.057692307978868484 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.10104166716337204 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4999999701976776 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2253139168024063 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5318312048912048 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9909475445747375 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904374480247498 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 +particle/sensitivity‚0.3208034336566925 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.11403509229421616 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_4_of_6‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.49892815947532654 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2302933782339096 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17905202507972717 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5166900753974915 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_1_of_6‚1 particle/accuracy_6_of_6‚1 particle/accuracy_2_of_7‚1 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1875 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9766522645950317 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.29970160126686096 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5552083253860474 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.041560158133506775 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5374054908752441 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12047021836042404 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1977006494998932 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.12939870357513428 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14055880904197693 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9913762211799622 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.45529866218566895 particle/accuracy_1_of_3‚1 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4135802090167999 jet/accuracy_3_of_7‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5129212737083435 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08164287358522415 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.2463723123073578 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6000000238418579 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9314420223236084 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5423842668533325 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.43153154850006104 *jet/accuracy_1_of_4‚0.9183846116065979 +jet/accuracy_3_of_4‚0.05931788682937622 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5082145929336548 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15318796038627625 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5051902532577515 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9980175495147705 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2669133245944977 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22456254065036774 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5276785492897034 +"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.75 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,jet/accuracy_4_of_4‚0.009432476945221424 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ."Purity/1FRt1SRqqt2FBt/event_purity‚0.03125 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.47035104036331177 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4660918712615967 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 *jet/accuracy_2_of_4‚0.3966118395328522 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12765957415103912 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1467435359954834 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.19230769574642181 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4464285671710968 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5149916410446167 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5216900706291199 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10867293179035187 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.43864333629608154 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4107142388820648 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.42666664719581604 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5138546228408813 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 (particle/accuracy‚0.8520068526268005 ;"Purity/3FRt0SRqqt*FBt/event_purity‚0.011111111380159855 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 jet/accuracy_5_of_5‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13798201084136963 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.32918480038642883 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_2‚1 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2916666567325592 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.41123512387275696 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 particle/accuracy_2_of_3‚1 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.36759260296821594 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5353296995162964 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5088996291160583 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 (particle/f_score‚0.40346530079841614 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_7‚0 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.48560962080955505 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5336117744445801 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5145819783210754 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 particle/accuracy_1_of_7‚1 particle/accuracy_6_of_7‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03359135240316391 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_5_of_5‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 particle/accuracy_3_of_7‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09734531491994858 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5165922045707703 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_7‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.49109095335006714 particle/accuracy_3_of_3‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9705422520637512 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17161333560943604 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6656745672225952 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4347745478153229 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9857953190803528 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.42659929394721985 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.46973374485969543 % _timestamp‚1.7885045019131505e+09 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5213015675544739 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.15948276221752167 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982480406761169 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.347423791885376 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1370670646429062 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4806201756000519 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.051313627511262894 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0694444477558136 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9919780492782593 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1654040515422821 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1187739446759224 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5083299875259399 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1614031046628952 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 epoch‚5 )loss/val_total_loss‚41.20612376517727 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9915932416915894 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9908289313316345 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.49269577860832214 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28703707456588745 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4591617286205292 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5789210796356201 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0023419205099344254 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14666666090488434 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27294421195983887 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.48470136523246765 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2064400613307953 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.25913164019584656 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 particle/accuracy_1_of_4‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5833333730697632 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5300322771072388 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15093407034873962 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9991645812988281 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7032302021980286 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 ."Purity/3FRt0SRqqt1FBt/event_purity‚0.03125 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4733295142650604 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.3333333134651184 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.031086791306734085 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16070248186588287 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7348485589027405 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 ,jet/accuracy_4_of_5‚0.004480286967009306 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49666666984558105 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1279793381690979 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.5555555820465088 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9909588098526001 particle/accuracy_2_of_6‚1 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.48164910078048706 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 *jet/accuracy_1_of_5‚0.9235087633132935 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.31481480598449707 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09568354487419128 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4481704831123352 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5126715302467346 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.49103713035583496 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.540123462677002 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3422027826309204 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07479183375835419 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4895833730697632 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5387672781944275 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9905832409858704 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1625974327325821 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 particle/accuracy_2_of_5‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9444444179534912 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 _runtime‚8559.692634945 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18063484132289886 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.487907350063324 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17369717359542847 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23732757568359375 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013307814486324787 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3472222089767456 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1875 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.030027486383914948 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0287477970123291 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.584016740322113 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5588499307632446 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1973457932472229 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_5‚1 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24765276908874512 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.991988480091095 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.27365103363990784 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14084552228450775 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19225622713565826 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1354166716337204 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 *jet/accuracy_2_of_6‚0.6025640964508057 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5833333730697632 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.01814058981835842 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.05143755301833153 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5690670609474182 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5712028741836548 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2138083577156067 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.26792457699775696 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9884798526763916 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.27586206793785095 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.4166666865348816 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1468253880739212 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5659502148628235 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15618190169334412 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998810350894928 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_3‚0.8750443458557129 particle/accuracy_3_of_4‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 trainer/global_step‚7253 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.40098005533218384 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.49698054790496826 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07350808382034302 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2712605595588684 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9635416865348816 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38639792799949646 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5217848420143127 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1614031046628952 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1°‰O ê ©¦ particle/accuracy_3_of_7‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09734531491994858 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5165922045707703 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_7‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.49109095335006714 particle/accuracy_3_of_3‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9705422520637512 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17161333560943604 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6656745672225952 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4347745478153229 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9857953190803528 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.42659929394721985 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.46973374485969543 % _timestamp‚1.7885045019131505e+09 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5213015675544739 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.15948276221752167 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/*ë6§ùFRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982480406761169 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.347423791885376 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1370670646429062 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4806201756000519 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.051313627511262894 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0694444477558136 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9919780492782593 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1654040515422821 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1187739446759224 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5083299875259399 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1614031046628952 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 epoch‚5 )loss/val_total_loss‚41.20612376517727 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9915932416915894 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9908289313316345 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.49269577860832214 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28703707456588745 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4591617286205292 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5789210796356201 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0023419205099344254 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14666666090488434 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27294421195983887 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.48470136523246765 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2064400613307953 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.25913164019584656 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 particle/accuracy_1_of_4‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5833333730697632 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5300322771072388 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15093407034873962 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9991645812988281 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7032302021980286 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 ."Purity/3FRt0SRqqt1FBt/event_purity‚0.03125 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4733295142650604 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.3333333134651184 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.031086791306734085 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16070248186588287 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7348485589027405 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 ,jet/accuracy_4_of_5‚0.004480286967009306 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49666666984558105 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1279793381690979 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.5555555820465088 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9909588098526001 particle/accuracy_2_of_6‚1 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.48164910078048706 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 *jet/accuracy_1_of_5‚0.9235087633132935 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.31481480598449707 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09568354487419128 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4481704831123352 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5126715302467346 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.49103713035583496 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.540123462677002 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3422027826309204 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07479183375835419 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4895833730697632 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5387672781944275 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9905832409858704 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1625974327325821 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 particle/accuracy_2_of_5‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9444444179534912 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 _runtime‚8559.692634945 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18063484132289886 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.487907350063324 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17369717359542847 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23732757568359375 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013307814486324787 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3472222089767456 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1875 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.030027486383914948 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0287477970123291 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.584016740322113 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5588499307632446 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1973457932472229 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_5‚1 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24765276908874512 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.991988480091095 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.27365103363990784 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14084552228450775 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19225622713565826 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1354166716337204 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 *jet/accuracy_2_of_6‚0.6025640964508057 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5833333730697632 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.01814058981835842 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.05143755301833153 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5690670609474182 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5712028741836548 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2138083577156067 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.26792457699775696 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9884798526763916 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.27586206793785095 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.4166666865348816 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1468253880739212 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5659502148628235 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15618190169334412 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998810350894928 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_3‚0.8750443458557129 particle/accuracy_3_of_4‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 trainer/global_step‚7253 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.40098005533218384 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.49698054790496826 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07350808382034302 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2712605595588684 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9635416865348816 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38639792799949646 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5217848420143127 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1614031046628952 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.49269577860832214 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960132241249084 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1963990330696106 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.15909092128276825 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 _step‚597 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22762346267700195 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_2_of_7‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41484448313713074 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16926893591880798 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993895292282104 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01662825606763363 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_5‚0.11773554980754852 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.017789427191019058 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.37995636463165283 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.11243592947721481 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.13245613873004913 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3872646391391754 *jet/accuracy_1_of_1‚0.4797857701778412 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5417895317077637 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16030772030353546 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9903751611709595 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4015151858329773 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6400061845779419 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19857648015022278 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1964285671710968 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0694444477558136 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.02923472598195076 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_6‚0.9807692170143127 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.38224396109580994 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.46666663885116577 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9926935434341431 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.4041666090488434 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4401519000530243 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5455459952354431 6 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.533991277217865 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1531531661748886 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.06378600746393204 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4907407760620117 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00877861212939024 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26229479908943176 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3982967734336853 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5350038409233093 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5096095204353333 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04459172487258911 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6929824948310852 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5327968597412109 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3472222089767456 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3860090374946594 +particle/specificity‚0.9502973556518555 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9771632552146912 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322916865348816 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14784832298755646 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3965167701244354 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3847711384296417 particle/accuracy_4_of_4‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 jet/accuracy_6_of_7‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8116438388824463 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9166666269302368 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 jet/accuracy_7_of_7‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.545976996421814 particle/accuracy_5_of_7‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1629781424999237 jet/accuracy_6_of_6‚0 particle/accuracy_5_of_6‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.030860714614391327 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5543498396873474 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9750000238418579 ;"Purity/3FRt*SRqqt1FBt/event_purity‚0.008333333767950535 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.593303918838501 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5006116032600403 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_2‚0.7531417012214661 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 *"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,jet/accuracy_4_of_6‚0.057692307978868484 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.10104166716337204 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4999999701976776 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2253139168024063 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5318312048912048 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9909475445747375 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904374480247498 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 +particle/sensitivity‚0.3208034336566925 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.11403509229421616 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_4_of_6‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.49892815947532654 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2302933782339096 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17905202507972717 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5166900753974915 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_1_of_6‚1 particle/accuracy_6_of_6‚1 particle/accuracy_2_of_7‚1 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1875 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9766522645950317 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.29970160126686096 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5552083253860474 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.041560158133506775 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5374054908752441 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12047021836042404 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1977006494998932 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.12939870357513428 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14055880904197693 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9913762211799622 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.45529866218566895 particle/accuracy_1_of_3‚1 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4135802090167999 jet/accuracy_3_of_7‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5129212737083435 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08164287358522415 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.2463723123073578 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6000000238418579 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9314420223236084 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5423842668533325 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.43153154850006104 *jet/accuracy_1_of_4‚0.9183846116065979 +jet/accuracy_3_of_4‚0.05931788682937622 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5082145929336548 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15318796038627625 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5051902532577515 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9980175495147705 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2669133245944977 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22456254065036774 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5276785492897034 +"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.75 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,jet/accuracy_4_of_4‚0.009432476945221424 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ."Purity/1FRt1SRqqt2FBt/event_purity‚0.03125 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.47035104036331177 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4660918712615967 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 *jet/accuracy_2_of_4‚0.3966118395328522 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12765957415103912 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1467435359954834 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.19230769574642181 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4464285671710968 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5149916410446167 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5216900706291199 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10867293179035187 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.43864333629608154 @&Purity/0FRt2SRqqt2FBt/event_proFŒL* portion‚0.0009920635493472219 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4107142388820648 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.42666664719581604 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5138546228408813 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 (particle/accuracy‚0.8520068526268005 ;"Purity/3FRt0SRqqt*FBt/event_purity‚0.011111111380159855 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 jet/accuracy_5_of_5‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13798201084136963 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.32918480038642883 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_2‚1 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2916666567325592 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.41123512387275696 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 particle/accuracy_2_of_3‚1 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.36759260296821594 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5353296995162964 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5088996291160583 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 (particle/f_score‚0.40346530079841614 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_7‚0 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.48560962080955505 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5336117744445801 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5145819783210754 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 particle/accuracy_1_of_7‚1 particle/accuracy_6_of_7‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03359135240316391 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_5_of_5‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875Õ£Rðqë ‚(:æ »ÓéÔÁÿÇm$ memory_percent‚93.01400184631348 cpu‚25.202758151561284 disk.rbd0.out‚ 2.466796875 disk.sdb1.out‚0( proc.memory.availableMB‚ 1144.5859375 disk.rbd2.in‚0 disk.md0.in‚ 4.18359375 disk.sda.out‚954.9892578125) proc.memory.percent‚22.34339714050293# disk./.usageGB‚2711.38920211792 disk.md0p1.out‚ 918.6484375 disk.sdb.out‚954.9892578125 disk.sda2.in‚ 3.55078125# disk.nvme0n1.out‚90714.97265625 disk.nvme1n1.in‚ 744.32421875 disk.sda1.in‚0 disk.sdb1.in‚0 disk.rbd2.out‚ 2.47900390625 network.recv‚ 2.805995e+06! disk.dm-0.out‚127985.02734375 disk.sda.in‚ 3.55078125 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1683.86328125 disk.sda2.out‚954.9892578125 disk.nvme0n1.in‚ 939.5390625" disk.nvme1n1.out‚ 37270.0546875 disk.md0.out‚ 918.6484375 network.sent‚ 1.8884959e+07 disk.sdb.in‚ 0.6328125 disk.sdb2.in‚ 0.6328125 disk.sdb2.out‚954.9892578125" proc.memory.rssMB‚ 3660.7421875( disk./.usagePercent‚75.8115768957587 disk.md0p1.in‚ 4.18359375‰“ûì ‚(:„ »ÓéÔÜóüm gpu.0.gpu‚5.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚32.0 gpu.0.powerWatts‚86.17' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚21.5425 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Žuˆ’í ‚(:‡ ÊÓéÔÆݸm gpu.0.gpu‚16.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚179.246' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.8115 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“™÷Âvî ‚(:ë ÊÓéÔÑ›òm disk.sda2.in‚ 3.55078125 disk.rbd0.out‚ 2.466796875 cpu‚25.201182497493175% disk./.usageGB‚2711.1372261047363 disk.nvme0n1.in‚ 939.5390625! disk.sdb2.out‚956.49365234375 disk.md0p1.out‚ 920.09375 network.sent‚ 1.9010556e+07) disk./.usagePercent‚75.80453154096993 disk.sdb.in‚ 0.6328125 disk.rbd2.in‚0 disk.sdb2.in‚ 0.6328125 disk.dm-0.in‚ 1683.86328125) proc.memory.availableMB‚ 1974.03515625# proc.memory.rssMB‚ 3004.48046875 disk.sda.out‚956.49365234375$ memory_percent‚87.95144557952881) proc.memory.percent‚18.33789348602295 proc.cpu.threads‚17! disk.sda2.out‚956.49365234375 disk.nvme1n1.in‚ 744.32421875 disk.rbd2.out‚ 2.47900390625 disk.rbd0.in‚0 disk.dm-0.out‚129731.7265625 disk.sda.in‚ 3.55078125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb.out‚956.49365234375# disk.nvme0n1.out‚92285.72265625 disk.md0p1.in‚ 4.18359375 network.recv‚ 2.812091e+06# disk.nvme1n1.out‚37446.00390625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.in‚ 4.18359375 disk.md0.out‚ 920.093759Èv‘ï ‚(:† ÙÓéÔ¤Ùœm gpu.0.gpu‚9.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.774' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.9435 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0LÊ¢í|ð ‚(:ñ ÙÓéÔ„Û†n disk.md0.in‚ 4.18359375 disk.dm-0.in‚ 1683.86328125% disk./.usageGB‚2711.1372261047363 disk.md0p1.out‚ 921.52734375! disk.dm-0.out‚129788.47265625 disk.sdb.in‚ 0.6328125! disk.sdb2.out‚957.99462890625 proc.cpu.threads‚17) disk./.usagePercent‚75.80453154096993 disk.rbd0.in‚0 disk.md0p1.in‚ 4.18359375! disk.sda2.out‚957.99462890625 disk.md0.out‚ 921.52734375) proc.memory.availableMB‚ 1973.87890625# proc.memory.rssMB‚ 3004.48046875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.nvme0n1.in‚ 939.5390625" disk.nvme0n1.out‚ 92285.9609375 disk.sda1.in‚0$ memory_percent‚87.95239925384521 disk.rbd2.in‚0 disk.sdb2.in‚ 0.6328125 cpu‚25.200918486998308 disk.nvme1n1.in‚ 744.32421875# disk.nvme1n1.out‚37502.51171875 disk.sda1.out‚0) proc.memory.percent‚18.33789348602295 disk.sda.in‚ 3.55078125 network.recv‚ 2.815613e+06 disk.rbd0.out‚ 2.466796875 disk.sdb.out‚957.99462890625 disk.rbd2.out‚ 2.47900390625 disk.sda.out‚957.99462890625 network.sent‚ 1.9015822e+07 disk.sda2.in‚ 3.55078125óhò`’ñ ‚(:‡ èÓéÔèéˆm gpu.0.gpu‚23.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zÇywò ‚(:ì èÓéÔ‡Þêm disk.md0p1.in‚ 4.18359375 disk.dm-0.in‚ 1683.86328125 disk.nvme0n1.in‚ 939.5390625 disk.nvme1n1.in‚ 744.32421875 disk.rbd2.out‚ 2.54150390625 disk.rbd0.out‚ 2.466796875 disk.sdb1.out‚0 network.sent‚ 1.9021944e+07 network.recv‚ 2.819864e+06) disk./.usagePercent‚75.78971084671056 disk.sda.out‚959.96826171875) proc.memory.percent‚18.33789348602295 disk.sdb.in‚ 0.6328125 disk.nvme1n1.out‚ 37622.46875 disk.md0.in‚ 4.18359375# proc.memory.rssMB‚ 3004.48046875 cpu‚25.20059974887332! disk.sda2.out‚959.96826171875 disk.rbd2.in‚0! disk.sdb2.out‚959.96826171875 disk.rbd0.in‚0$ disk./.usageGB‚2710.607166290283# disk.nvme0n1.out‚92443.19921875 disk.sda1.in‚0 disk.sdb1.in‚0 disk.md0.out‚ 923.44140625 disk.sdb.out‚959.96826171875 disk.sda2.in‚ 3.55078125 disk.sdb2.in‚ 0.6328125 disk.sda.in‚ 3.55078125 proc.cpu.threads‚17 disk.md0p1.out‚ 923.44140625! disk.dm-0.out‚130065.66796875 disk.sda1.out‚0# memory_percent‚87.9535436630249) proc.memory.availableMB‚ 1973.69140625CÌä2ó - _wandbruntime‚8617  _runtime‚8617|ô †Éô à trainer/global_step‚7299 % _timestamp‚1.7885045598126185e+09 _runtime‚8617.589949106 _step‚598 ' lr-AdamW/pg1‚0.00042808260807002027 ' lr-AdamW/pg2‚0.00042808260807002027WÞD¸Îõ È ' lr-AdamW/pg1‚0.00042808260807002027 ' lr-AdamW/pg2‚0.00042808260807002027 trainer/global_step‚7299 % _timestamp‚1.7885045598126185e+09 _runtime‚8617.589949106 _step‚598ÖÚg;2ö - _wandbruntime‚8617  _runtime‚8617GÒ$vû'÷ õO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚679937792181760 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚595911359585936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚679937768241664 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8373662730752 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚624727450894336 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8746983848960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 185088495 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚624728378868736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚613 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 % _timestamp‚1.7885045598134317e+09 _step‚599 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚616353788163584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91695986 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚688685011104256 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚595911342546576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153028720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185114839 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚604254626053922 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8343266467986 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 185114839 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _runtime‚ 8617.59080767 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚624728378868736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚616353788163584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235073536 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 136128357 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8374572702208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 338143561 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8374572702208 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259013632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚624727450894336 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 338117215 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8747218922496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153028722 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8342358492982 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8373662730752 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚611 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 136127744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91695984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚624727450894336 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8746983848960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 185088495 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚624728378868736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚613 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 % _timestamp‚1.7885045598134317e+09 _step‚599 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚616353788163584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91695986 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚688685011104256 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚595911342546576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153028720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185114839 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚604254626053922 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8343266467986 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 185114839 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _runtime‚ 8617.59080767 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚624728378868736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚616353788163584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235073536 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 136128357 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8374572702208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 338143561 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8374572702208 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259013632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚624727450894336 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 338117215 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8747218922496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153028722 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8342358492982 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8373662730752 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚611 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 136127744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91695984×HÊl2ù - _wandbruntime‚8618  _runtime‚8618,kÂÉ ú ÿM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8375694175744 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚7299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚624811119428096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136145544 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚604334760921952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 338162206 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885045609316525e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 338188578 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185139696 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚604333835907380 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44437495 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚600 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153048880 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91708049 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44438099 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚616435425252352 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91708051 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8748142275584 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚595990360141120 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8748377335808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153048882 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 338162206 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 185113326 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 338188578 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8375694175744 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚624811119428096 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚624810191440384 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚616435425252352 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8344383741472 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144±^º0 'û „N WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚616435407249408 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153048880 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136145544 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚604334760921952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 338162206 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885045609316525e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 338188578 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185139696 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚604333835907380 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44437495 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚600 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153048880 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91708049 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44438099 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚616435425252352 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91708051 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8748142275584 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚595990360141120 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8748377335808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153048882 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 338162206 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 185113326 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 338188578 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8375694175744 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚624811119428096 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚624810191440384 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚616435425252352 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8344383741472 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8375694175744 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚7299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚624811119428096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44715914 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚628864492610560 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 186333397 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8805393201152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStat!:ÚG]sMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚8675.378928243 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 340370117 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚608261383036230 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8430660127232 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 186359741 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚603 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8398235748090 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 186333397 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8429750155776 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚628865420584960 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚620434742454784 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚693254697290240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚92299234 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚684449053464064 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚628865420584960 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚620434742454784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 % _timestamp‚1.7885046176010542e+09Kbo(Œ!ûO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚628865420584960 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚620434742454784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 % _timestamp‚1.7885046176010542e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154036722 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8399143723094 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 340396463 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8430660127232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚599862222273776 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44715914 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚628864492610560 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 186333397 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8805393201152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚8675.378928243 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 340370117 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚608261383036230 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8430660127232 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 186359741 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚603 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8398235748090 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 186333397 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8429750155776 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚628865420584960 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚620434742454784 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚693254697290240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚92299234 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚684449053464064 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640ÛÊœr"2!- _wandbruntime‚8676  _runtime‚8676E~I/'Ž!þM H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8431781600768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 trainer/global_step‚7349 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚608340592889688 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚628947233156608 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚608341517904260 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚92311301 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8431781600768 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 340441480 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 137033419 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚620516361540608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44722118 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚599941239868320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚628948161144320 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 186384598 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 340415108 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚620516379543552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚620516361540608 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚620516379543552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154056882 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 186358228 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚693345836729344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚684539303108608 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8400260996580 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 _step‚604 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92311299 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚693346091535360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8430871616000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 340441480 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8399353021368 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137032799 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8806788426752 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚599941256907680 # _timestamp‚1.78850461861489e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154056880 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚628948161144320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0ÛÏ ÅI!ƒN VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 186358228 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚620516361540608 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚620516379543552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154056882 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 186358228 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚693345836729344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚684539303108608 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8400260996580 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 _step‚604 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92311299 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚693346091535360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8430871616000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 340441480 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8399353021368 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137032799 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8806788426752 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚599941256907680 # _timestamp‚1.78850461861489e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154056880 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚628948161144320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8431781600768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 trainer/global_step‚7349 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚608340592889688 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚628947233156608 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚608341517904260 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚92311301 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8431781600768 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 340441480 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 137033419 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚620516361540608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44722118 E?DeviceStatsMonitor.on_train_batch_eQÄCÇÀnd/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚599941239868320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚628948161144320 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 186384598 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 340415108 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚620516379543552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚633001534326784 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8486747552256 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 342623019 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚697824134574080 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚688960362626560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚624515714748928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚603813102000976 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8454113003198 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚697824378907648 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚603813119040336 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 187578299 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚612268140018538 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8486747552256 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137905454 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8863795887616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45002968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 $ _timestamp‚1.788504674891951e+09 _runtime‚8732.669277866 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2Û¿ö( !úO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚633002462301184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155044720 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚612267215004174 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚624515714748928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚603813102000976 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8454113003198 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚697824378907648 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚603813119040336 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 187578299 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚612268140018538 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8486747552256 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137905454 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8863795887616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45002968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 $ _timestamp‚1.788504674891951e+09 _runtime‚8732.669277866 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚605 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚92902486 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚624515696745984 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8485837580800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚624515696745984 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 187604643 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚92902484 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45002363 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚633001534326784 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8486747552256 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 342623019 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚697824134574080 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚688960362626560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 JBDeviceStatsMonitor.on_train_batch_end/seg•ãSRg$ment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8487869025792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92914549 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 342668010 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 155064882 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8864949121536 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 $ _timestamp‚1.788504676020423e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8865186278912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 137923259 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8456138251688 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8487869025792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8455230276476 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚612348274886568 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 187603130 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚92914551 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚689050564390912 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137922637 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚603892136634880 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155064880 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚624597315831808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚633085202860544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 155064880 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 _step‚608 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8486959041024 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 342694382 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 trainer/global_step‚7399 _runtime‚8733.798324097 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342694382 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚603892119595520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8456138251688 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8487869025792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8455230276476 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚612348274886568 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 187603130 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚92914551 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚689050564390912 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137922637 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚603892136634880 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155064880 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚624597315831808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚633085202860544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 155064880 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 _step‚608 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8486959041024 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 342694382 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 trainer/global_step‚7399 _runtime‚8733.798324097 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342694382 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚603892119595520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8487869025792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92914549 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 342668010 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 155064882 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8864949121536ऋ\Ý2¤!- _wandbruntime‚8733  _runtime‚8733vöW¥!Ñ 0loss/SRqqt4/detection_loss‚2.240059223867232 0loss/FRt2/assignment_loss‚3.2671025594075522 1loss/SRqqt1/assignment_loss‚1.341562009382027 1loss/FBt2/assignment_loss‚0.21269750436476897 /loss/FRt3/detection_loss‚1.6678956559435887 -loss/FRt1/detection_loss‚2.00998727611376 %loss/total_loss‚43.40664363875288 epoch‚6 _runtime‚8733.799329602 % _timestamp‚1.7885046760222988e+09 /loss/FRt4/assignment_loss‚3.415333148204919 1loss/SRqqt2/assignment_loss‚1.608505716919899 0loss/FBt1/assignment_loss‚0.1058045169939578 0loss/FBt3/assignment_loss‚0.2052176452416461 1loss/SRqqt1/detection_loss‚1.9298525154590607 1loss/SRqqt3/detection_loss‚0.9059643641646419 2loss/SRqqt4/assignment_loss‚1.5859150451517874 /loss/FBt2/detection_loss‚2.2660922358433404 .loss/FBt3/detection_loss‚3.419118896126747 _step‚609 0loss/FRt3/assignment_loss‚1.2143608277494258 /loss/FRt2/detection_loss‚2.3809151072942076 .loss/FRt4/detection_loss‚3.004146285129316 /loss/FBt1/detection_loss‚1.0708715742302592 2loss/SRqqt3/assignment_loss‚0.8824509719905881 1loss/FBt4/assignment_loss‚0.15182250563455568 1loss/SRqqt2/detection_loss‚2.6518006134033203 .loss/FBt4/detection_loss‚2.797543451189995 trainer/global_step‚7399 /loss/FRt1/assignment_loss‚3.071623988946279Šè Ñ\¦!Ö 0loss/FBt3/assignment_loss‚0.2052176452416461 1loss/SRqqt1/detection_loss‚1.9298525154590607 1loss/SRqqt3/detection_loss‚0.9059643641646419 2loss/SRqqt4/assignment_loss‚1.5859150451517874 /loss/FBt2/detection_loss‚2.2660922358433404 .loss/FBt3/detection_loss‚3.419118896126747 _step‚609 0loss/FRt3/assignment_loss‚1.2143608277494258 /loss/FRt2/detection_loss‚2.3809151072942076 .loss/FRt4/detection_loss‚3.004146285129316 /loss/FBt1/detection_loss‚1.0708715742302592 2loss/SRqqt3/assignment_loss‚0.8824509719905881 1loss/FBt4/assignment_loss‚0.15182250563455568 1loss/SRqqt2/detection_loss‚2.6518006134033203 .loss/FBt4/detection_loss‚2.797543451189995 trainer/global_step‚7399 /loss/FRt1/assignment_loss‚3.071623988946279 0loss/SRqqt4/detection_loss‚2.240059223867232 0loss/FRt2/assignment_loss‚3.2671025594075522 1loss/SRqqt1/assignment_loss‚1.341562009382027 1loss/FBt2/assignment_loss‚0.21269750436476897 /loss/FRt3/detection_loss‚1.6678956559435887 -loss/FRt1/detection_loss‚2.00998727611376 %loss/total_loss‚43.40664363875288 epoch‚6 _runtime‚8733.799329602 % _timestamp‚1.7885046760222988e+09 /loss/FRt4/assignment_loss‚3.415333148204919 1loss/SRqqt2/assignment_loss‚1.608505716919899 0loss/FBt1/assignment_loss‚0.1058045169939578á«ý½m’§!‚(:‡ ïÔéÔ¦¡ƒm gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚179.246' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.8115 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0PjÉs¨!‚(:è ïÔéÔæ¼ém disk.nvme1n1.in‚ 744.32421875 disk.md0.in‚ 4.18359375 disk.sdb2.in‚ 0.6328125 disk.sda1.in‚0 disk.sdb.out‚974.96337890625! disk.nvme0n1.out‚ 94460.203125# disk.nvme1n1.out‚37625.67578125 disk.md0.out‚ 937.9140625 disk.rbd2.in‚0 disk.rbd2.out‚ 2.69140625 disk.sda.out‚974.96337890625 disk.sda2.in‚ 3.55078125 disk.nvme0n1.in‚ 939.5390625! disk.dm-0.out‚132085.87890625# proc.memory.rssMB‚ 3004.48046875) proc.memory.percent‚18.33789348602295 proc.cpu.threads‚17 disk.sda.in‚ 3.55078125 disk.sda1.out‚0 disk.sdb1.out‚0 disk.md0p1.in‚ 4.18359375 network.sent‚ 1.9275686e+07 network.recv‚ 2.859259e+06( disk./.usagePercent‚75.7897040204365! disk.sda2.out‚974.96337890625 disk.md0p1.out‚ 937.9140625 disk.sdb1.in‚0! disk.sdb2.out‚974.96337890625 disk.dm-0.in‚ 1683.86328125# memory_percent‚87.9512071609497 cpu‚25.197911109601595$ disk./.usageGB‚2710.606922149658 disk.sdb.in‚ 0.6328125 disk.rbd0.in‚0 disk.rbd0.out‚ 2.466796875) proc.memory.availableMB‚ 1974.07421875fÚØò“©!‚(:ˆ þÔéÔ¿ôÃm gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.333' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.33325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄÎIbª!‚(:× þÔéÔÕª€n disk.nvme1n1.in‚ 744.32421875! disk.nvme0n1.out‚ 94618.046875 disk.md0.in‚ 4.2890625 disk.sda1.in‚0# proc.memory.rssMB‚ 3004.48046875 proc.cpu.threads‚17 network.sent‚ 1.9383645e+07 disk.rbd0.in‚0 disk.rbd0.out‚ 2.466796875 disk.sdb.out‚977.1298828125 disk.sda2.in‚ 3.6171875 disk.nvme1n1.out‚ 37625.875 disk.rbd2.out‚ 2.69140625 disk.sdb2.in‚0.671875 disk.md0p1.in‚ 4.2890625 disk.nvme0n1.in‚ 939.5390625 disk.md0.out‚ 940.0078125 disk.rbd2.in‚0 disk.sda1.out‚0 cpu‚25.19762900510574% disk./.usageGB‚2710.5562705993652 disk.sdb1.in‚0 disk.sda.in‚ 3.6171875 disk.sda.out‚977.1298828125) proc.memory.percent‚18.33789348602295 network.recv‚ 2.865202e+06 disk.sdb1.out‚0 disk.dm-0.in‚ 1683.86328125 disk.sda2.out‚977.1298828125$ memory_percent‚87.95270919799805 disk.sdb2.out‚977.1298828125) disk./.usagePercent‚75.78828778189099 disk.dm-0.out‚ 132243.921875 disk.sdb.in‚0.671875' proc.memory.availableMB‚ 1973.828125 disk.md0p1.out‚ 940.0078125ª ·“«!‚(:ˆ ÕéÔ¹œûl gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.395' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.09875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|¬¹pd¬!‚(:Ù ÕéÔÙ°„n! disk.dm-0.out‚132586.87109375 disk.nvme0n1.in‚ 939.5390625 disk.rbd2.out‚ 2.69140625$ memory_percent‚91.96844100952148 cpu‚25.19732038668337) disk./.usagePercent‚75.79055399821698$ disk./.usageGB‚2710.637321472168 disk.sdb.in‚0.671875 disk.nvme1n1.in‚ 744.32421875# proc.memory.rssMB‚ 3004.48046875) proc.memory.percent‚18.33789348602295 disk.sdb.out‚978.6435546875 proc.cpu.threads‚17 disk.sda2.in‚ 3.6171875# disk.nvme1n1.out‚37626.12109375 disk.sda.in‚ 3.6171875 disk.sda1.in‚0 disk.rbd0.in‚0 network.recv‚ 2.8687e+06 disk.md0.in‚ 4.2890625 disk.md0.out‚ 941.45703125 disk.sda.out‚978.6435546875 disk.sda1.out‚0 disk.sdb1.in‚0 disk.sda2.out‚978.6435546875 disk.nvme0n1.out‚94960.75 disk.rbd2.in‚0 disk.sdb2.in‚0.671875 disk.rbd0.out‚ 2.466796875' proc.memory.availableMB‚ 1315.890625 disk.md0p1.in‚ 4.2890625 disk.sdb2.out‚978.6435546875 disk.sdb1.out‚0 network.sent‚ 1.9391101e+07 disk.md0p1.out‚ 941.45703125 disk.dm-0.in‚ 1683.86328125wa]!­!‚(:„ œÕéÔ–æ‡m gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚125.88' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.47 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØšGöp®!‚(:å œÕéÔŽån disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 1315.52734375) disk./.usagePercent‚75.79405875664197! disk.dm-0.out‚132828.41796875" disk.nvme0n1.out‚ 95202.1484375 disk.rbd2.in‚0 disk.rbd2.out‚ 2.69140625$ memory_percent‚91.97065830230713 disk.sdb.out‚980.08740234375 disk.md0.out‚ 942.8359375# proc.memory.rssMB‚ 3004.48046875 cpu‚25.197012541953814 proc.cpu.threads‚17 network.recv‚ 2.873178e+06 disk.rbd0.out‚ 2.466796875 disk.sdb.in‚0.671875 disk.sda2.in‚ 3.6171875 disk.nvme0n1.in‚ 939.5390625 disk.nvme1n1.in‚ 744.32421875 disk.md0.in‚ 4.2890625 disk.sda.in‚ 3.6171875 disk.md0p1.in‚ 4.2890625! disk.sdb2.out‚980.08740234375 disk.sda1.out‚0 disk.md0p1.out‚ 942.8359375! disk.sda2.out‚980.08740234375 disk.sda.out‚980.08740234375 network.sent‚ 1.9397197e+07$ disk./.usageGB‚2710.762668609619 disk.dm-0.in‚ 1683.86328125 disk.sdb2.in‚0.671875) proc.memory.percent‚18.33789348602295 disk.rbd0.in‚0# disk.nvme1n1.out‚37626.26953125?m[;2¯!- _wandbruntime‚8790  _runtime‚8790†† AÉ°!à _runtime‚8790.037223155 _step‚610 ' lr-AdamW/pg1‚0.00040579320192424864 ' lr-AdamW/pg2‚0.00040579320192424864 trainer/global_step‚7449 % _timestamp‚1.7885047322598877e+09£+!åα!È _step‚610 ' lr-AdamW/pg1‚0.00040579320192424864 ' lr-AdamW/pg2‚0.00040579320192424864 trainer/global_step‚7449 % _timestamp‚1.7885047322598877e+09 _runtime‚8790.037223155âÀE¨Ý2²!- _wandbruntime‚8790  _runtime‚8790®5?QA³!öO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚607763981728176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 344875921 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8542834977280 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚702393686844416 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚693471647848960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156052720 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8509990258306 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚8790.038542871 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 344902267 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156052722 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156052722 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚619 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45288031 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚607763998767536 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚621 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138793765 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚637138576043008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4528Aj8ž»8650 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8541925005824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚693471623908864 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8922062935552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8542834977280 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242236416 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚637139504017408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚702393929080832 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 % _timestamp‚1.7885047322609694e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8510898233310 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 trainer/global_step‚7449 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93505736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218296320 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚637139504017408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚702393929080832 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 % _timestamp‚1.7885047322609694e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8510898233310 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 trainer/global_step‚7449 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93505736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218296320 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156052722 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156052722 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚619 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45288031 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚607763998767536 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚621 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138793765 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚637138576043008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45288650 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8541925005824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚693471623908864 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8922062935552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8542834977280 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242236416 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144ãë¢Õ2µ!- _wandbruntime‚8791  _runtime‚8791å{F©'¶!€N XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8543046466048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 138812203 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45294402 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188848032 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 344947284 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156072882 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚702485335319552 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885047334313798e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚702485084707840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚693561873553408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚637222244576768 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚612 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚625 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 188874402 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45293779 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156072880 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚7449 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚616354106854304 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188874402 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138811578 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚607842999322720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚693561849613312 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 188848032 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712…gvé·!…N QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚693561873553408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚637222244576768 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚612 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚625 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 188874402 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45293779 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156072880 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚7449 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚616354106854304 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188874402 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138811578 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚607842999322720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚693561849613312 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 188848032 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8543046466048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 138812203 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45294402 D>DeviceáXt"StatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188848032 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 344947284 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156072882 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚702485335319552 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885047334313798e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚702485084707840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2äi{î«2¸!- _wandbruntime‚8791  _runtime‚8791µÆ„T¹!Î 0loss/FRt3/assignment_loss‚1.5359783034393752 0loss/FBt3/assignment_loss‚0.1167345006542746 epoch‚6 trainer/global_step‚7449 /loss/FRt4/assignment_loss‚2.849235103048127 1loss/SRqqt2/assignment_loss‚1.882728645950556 /loss/FBt1/detection_loss‚0.7731255493666707 .loss/FBt2/detection_loss‚2.142578116485051 /loss/FBt3/detection_loss‚3.2516065910458565 2loss/SRqqt3/assignment_loss‚1.1391380400504028 0loss/FBt4/assignment_loss‚0.7652962837415122 /loss/FRt1/detection_loss‚2.3336565909438116 .loss/FRt4/detection_loss‚2.780467251251484 1loss/SRqqt1/detection_loss‚2.1016871164242428 $loss/total_loss‚43.5282200466594 _step‚613 .loss/FRt2/detection_loss‚2.503719165814774 1loss/SRqqt4/detection_loss‚2.4473364365597567 2loss/SRqqt4/assignment_loss‚1.6917819579442341 .loss/FBt4/detection_loss‚2.010889178940228 _runtime‚8791.211688999 .loss/FRt1/assignment_loss‚2.84531792146819 2loss/SRqqt1/assignment_loss‚1.4757495736082396 0loss/SRqqt2/detection_loss‚3.043925125896931 /loss/FRt2/assignment_loss‚2.996702320235116 0loss/FBt1/assignment_loss‚0.1931175600064808 1loss/FBt2/assignment_loss‚0.16679640002985252 1loss/SRqqt3/detection_loss‚1.0117482042714823 % _timestamp‚1.7885047334344625e+09 .loss/FRt3/detection_loss‚1.468904109482748éG±Yº!Ó 2loss/SRqqt4/assignment_loss‚1.6917819579442341 .loss/FBt4/detection_loss‚2.010889178940228 _runtime‚8791.211688999 .loss/FRt1/assignment_loss‚2.84531792146819 2loss/SRqqt1/assignment_loss‚1.4757495736082396 0loss/SRqqt2/detection_loss‚3.043925125896931 /loss/FRt2/assignment_loss‚2.996702320235116 0loss/FBt1/assignment_loss‚0.1931175600064808 1loss/FBt2/assignment_loss‚0.16679640002985252 1loss/SRqqt3/detection_loss‚1.0117482042714823 % _timestamp‚1.7885047334344625e+09 .loss/FRt3/detection_loss‚1.468904109482748 0loss/FRt3/assignment_loss‚1.5359783034393752 0loss/FBt3/assignment_loss‚0.1167345006542746 epoch‚6 trainer/global_step‚7449 /loss/FRt4/assignment_loss‚2.849235103048127 1loss/SRqqt2/assignment_loss‚1.882728645950556 /loss/FBt1/detection_loss‚0.7731255493666707 .loss/FBt2/detection_loss‚2.142578116485051 /loss/FBt3/detection_loss‚3.2516065910458565 2loss/SRqqt3/assignment_loss‚1.1391380400504028 0loss/FBt4/assignment_loss‚0.7652962837415122 /loss/FRt1/detection_loss‚2.3336565909438116 .loss/FRt4/detection_loss‚2.780467251251484 1loss/SRqqt1/detection_loss‚2.1016871164242428 $loss/total_loss‚43.5282200466594 _step‚613 .loss/FRt2/detection_loss‚2.503719165814774 1loss/SRqqt4/detection_loss‚2.4473364365597567åø÷3’»!‚(:‡ «ÕéÔçâÅn gpu.0.gpu‚45.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.118' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.0295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§ªVe¼!‚(:Ú «ÕéÔä …o disk.sdb2.in‚0.671875 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 1.9413642e+07% disk./.usageGB‚2710.8873329162598 disk.sdb.out‚982.2626953125 disk.sda2.in‚ 3.6171875 disk.rbd2.out‚ 2.69140625) proc.memory.percent‚18.33789348602295 disk.sdb1.in‚0 disk.sda.in‚ 3.6171875 disk.sdb1.out‚0 disk.nvme0n1.in‚ 939.5390625 disk.nvme1n1.in‚ 744.32421875 disk.md0.out‚944.9375 disk.sdb2.out‚982.2626953125 disk.md0p1.out‚944.9375 disk.sdb.in‚0.671875$ memory_percent‚91.97094440460205 cpu‚25.19675723023137 disk.sda.out‚982.2626953125 disk.rbd0.in‚0 disk.rbd0.out‚ 2.466796875 disk.dm-0.in‚ 1683.86328125 disk.dm-0.out‚ 132975.484375# disk.nvme0n1.out‚95348.91015625# disk.nvme1n1.out‚37626.57421875 disk.md0.in‚ 4.2890625 network.recv‚ 2.877304e+06) disk./.usagePercent‚75.79754442283169 disk.sda1.out‚0 disk.md0p1.in‚ 4.2890625 disk.sda2.out‚982.2626953125 disk.rbd2.in‚0) proc.memory.availableMB‚ 1315.48046875# proc.memory.rssMB‚ 3004.48046875ÕRx’½!‚(:‡ ºÕéÔ–ìýl gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.774' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.9435 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ƚ¸q¾!‚(:æ ºÕéÔ¦ùÞm disk.md0.out‚ 946.25390625 disk.rbd0.in‚0 disk.dm-0.in‚ 1683.86328125! disk.nvme1n1.out‚ 37626.734375) proc.memory.percent‚18.33789348602295 disk.md0.in‚ 4.2890625 disk.rbd2.in‚0 disk.rbd2.out‚ 2.69140625 disk.sda.in‚ 3.6171875 disk.sdb2.in‚0.671875! disk.sdb2.out‚983.63916015625 disk.sda1.out‚0 disk.rbd0.out‚ 2.466796875 disk.nvme0n1.in‚ 939.5390625) disk./.usagePercent‚75.80453399416217 disk.sdb1.out‚0 disk.sdb.out‚983.63916015625 disk.sda2.in‚ 3.6171875# proc.memory.rssMB‚ 3004.48046875 network.recv‚ 2.883253e+06 disk.sdb1.in‚0 disk.md0p1.in‚ 4.2890625! disk.sda2.out‚983.63916015625 disk.nvme1n1.in‚ 744.32421875$ memory_percent‚91.97134971618652 network.sent‚ 1.9511116e+07 disk.sda1.in‚0 disk.md0p1.out‚ 946.25390625# disk.nvme0n1.out‚95646.46484375 cpu‚25.196459594711406 disk.sda.out‚983.63916015625% disk./.usageGB‚2711.1373138427734! disk.dm-0.out‚133273.19921875 disk.sdb.in‚0.671875( proc.memory.availableMB‚ 1315.4140625 proc.cpu.threads‚17÷§ÌP“¿!‚(:ˆ ÉÕéԧʟm gpu.0.gpu‚52.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚120.927' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.23175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñoÀ!‚(:ä ÉÕéÔ§õÛm disk.rbd0.out‚ 2.466796875 disk.md0p1.out‚ 947.8359375 disk.dm-0.out‚ 133274.03125 disk.nvme0n1.in‚ 939.5390625$ memory_percent‚91.96970462799072% disk./.usageGB‚2711.1373138427734# disk.nvme0n1.out‚95646.84765625 network.recv‚ 2.886076e+06! disk.sda2.out‚985.27880859375) proc.memory.availableMB‚ 1315.68359375 disk.sdb1.out‚0 disk.dm-0.in‚ 1683.86328125# proc.memory.rssMB‚ 3004.48046875 network.sent‚ 1.9513253e+07 disk.rbd2.out‚ 2.69140625 disk.sdb1.in‚0 disk.sda2.in‚ 3.6171875 cpu‚25.196183563371115 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.md0p1.in‚ 4.2890625) proc.memory.percent‚18.33789348602295) disk./.usagePercent‚75.80453399416217 disk.md0.in‚ 4.2890625! disk.sdb2.out‚985.27880859375 disk.sda.in‚ 3.6171875 disk.sdb.in‚0.671875 disk.md0.out‚ 947.8359375 disk.sdb2.in‚0.671875 disk.rbd0.in‚0 disk.sdb.out‚985.27880859375 disk.nvme1n1.in‚ 744.32421875# disk.nvme1n1.out‚37627.18359375 disk.sda.out‚985.27880859375 disk.sda1.in‚0 disk.sda1.out‚0·u_2Á!- _wandbruntime‚8847  _runtime‚8847ÓŸooÇÂ!Á & lr-AdamW/pg1‚0.0003984115157103546 & lr-AdamW/pg2‚0.0003984115157103546 trainer/global_step‚7499 % _timestamp‚1.7885047900513337e+09 _runtime‚8847.828798296 _step‚614³¢ÊºÌÃ!Æ _runtime‚8847.828798296 _step‚614 & lr-AdamW/pg1‚0.0003984115157103546 & lr-AdamW/pg2‚0.0003984115157103546 trainer/global_step‚7499 % _timestamp‚1.7885047900513337e+09æŠ.Þ€2Ä!- _wandbruntime‚8847  _runtime‚8847HVZü'Å!öO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚632677605328384 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8598012430848 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 347155169 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚697982933071360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157060722 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 190068103 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 % _timestamp‚1.7885047900525975e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8598922402304 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8980516907520 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚620281653983154 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚632677605328384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚8847.830173233 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚612 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚706963668275200 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚697982909131264 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 139683760 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 347128823 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157060720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157060720 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚641275617759232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8598012430848 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚706963426038784 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 190094447 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139683148 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45574164 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚611714878494736 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚641275617759232 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚632677623331328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94108984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚615AzL—(Æ!ûO UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157060722 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8980735203840 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 347128823 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157060720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157060720 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚641275617759232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8598012430848 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚706963426038784 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 190094447 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139683148 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45574164 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚611714878494736 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚641275617759232 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚632677623331328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94108984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚632677605328384 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8598012430848 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 347155169 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚697982933071360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157060722 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 190068103 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 % _timestamp‚1.7885047900525975e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8598922402304 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8980516907520 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚620281653983154 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚632677605328384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚8847.830173233 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚612 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚706963668275200 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚697982909131264 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 139683760 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_baGŠý§%tch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157080882 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45579749 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚698073134835712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 _runtime‚8848.807097874 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚641359286292992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚632759224414208 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚620361788851184 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚632759224414208 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157080880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _step‚616 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚707055067172352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚707054814463488 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚641358358305280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚620360863836612 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139701410 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45580359 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 190092934 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8599133891072 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 190119304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94121051 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157080880 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347200186 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139700798 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚610 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8566984786692 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 347200186 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 190119304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 347173814 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8600043875840 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 190092934 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚698073158775808 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0£)´› 'É!…N E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94121049 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157080882 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45579749 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚698073134835712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 _runtime‚8848.807097874 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚641359286292992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚632759224414208 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚620361788851184 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚632759224414208 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157080880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _step‚616 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚707055067172352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚707054814463488 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚641358358305280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚620360863836612 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139701410 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45580359 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 190092934 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8599133891072 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 190119304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94121051 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157080880 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347200186 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139700798 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚610 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8566984786692 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 347200186 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 190119304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 347173814 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8600043875840 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 190092934 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚698073158775808 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393èÌÀ+X2Ê!- _wandbruntime‚8848  _runtime‚8848ɱ—YË!Ó 0loss/SRqqt3/detection_loss‚0.896452792319319 0loss/SRqqt4/detection_loss‚2.865001282095909 trainer/global_step‚7499 0loss/FRt1/assignment_loss‚2.3915832061921396 0loss/FRt3/assignment_loss‚1.2525227423441612 .loss/FRt1/detection_loss‚2.057369656327571 /loss/FBt1/detection_loss‚0.5847548893911557 .loss/FBt2/detection_loss‚2.035279219641405 % _timestamp‚1.7885047910316873e+09 /loss/FRt2/assignment_loss‚3.232908080683814 1loss/SRqqt2/assignment_loss‚1.647848326638341 .loss/FRt2/detection_loss‚2.272467599891954 0loss/SRqqt1/detection_loss‚1.915400757143895 /loss/FBt4/detection_loss‚1.9217745278562819 _runtime‚8848.808689931 2loss/SRqqt1/assignment_loss‚1.3948968822757404 2loss/SRqqt3/assignment_loss‚0.9751404664148727 epoch‚6 _step‚617 &loss/total_loss‚42.639264973130494 1loss/FBt2/assignment_loss‚0.26094453292556435 1loss/FBt4/assignment_loss‚0.45118212679679864 /loss/FRt3/detection_loss‚1.6970821571047023 .loss/FRt4/detection_loss‚3.079728307723999 0loss/SRqqt2/detection_loss‚2.466074526309967 0loss/FRt4/assignment_loss‚3.1138124799728395 2loss/SRqqt4/assignment_loss‚1.8975467436015605 /loss/FBt3/detection_loss‚3.6932520866394043 0loss/FBt1/assignment_loss‚0.1424322651813744 1loss/FBt3/assignment_loss‚0.393809317657724031ÈHÐ^Ì!Ø 0loss/SRqqt1/detection_loss‚1.915400757143895 /loss/FBt4/detection_loss‚1.9217745278562819 _runtime‚8848.808689931 2loss/SRqqt1/assignment_loss‚1.3948968822757404 2loss/SRqqt3/assignment_loss‚0.9751404664148727 epoch‚6 _step‚617 &loss/total_loss‚42.639264973130494 1loss/FBt2/assignment_loss‚0.26094453292556435 1loss/FBt4/assignment_loss‚0.45118212679679864 /loss/FRt3/detection_loss‚1.6970821571047023 .loss/FRt4/detection_loss‚3.079728307723999 0loss/SRqqt2/detection_loss‚2.466074526309967 0loss/FRt4/assignment_loss‚3.1138124799728395 2loss/SRqqt4/assignment_loss‚1.8975467436015605 /loss/FBt3/detection_loss‚3.6932520866394043 0loss/FBt1/assignment_loss‚0.1424322651813744 1loss/FBt3/assignment_loss‚0.39380931765772403 0loss/SRqqt3/detection_loss‚0.896452792319319 0loss/SRqqt4/detection_loss‚2.865001282095909 trainer/global_step‚7499 0loss/FRt1/assignment_loss‚2.3915832061921396 0loss/FRt3/assignment_loss‚1.2525227423441612 .loss/FRt1/detection_loss‚2.057369656327571 /loss/FBt1/detection_loss‚0.5847548893911557 .loss/FBt2/detection_loss‚2.035279219641405 % _timestamp‚1.7885047910316873e+09 /loss/FRt2/assignment_loss‚3.232908080683814 1loss/SRqqt2/assignment_loss‚1.647848326638341 .loss/FRt2/detection_loss‚2.272467599891954é<‰j“Í!‚(:ˆ ØÕéÔו½l gpu.0.gpu‚39.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚177.071' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.26775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04Õ2ÓgÎ!‚(:Ü ØÕéÔýÄêl disk.sda1.out‚0 disk.nvme1n1.in‚ 744.32421875 disk.sdb.out‚ 986.740234375 disk.md0.in‚ 4.2890625 network.recv‚ 2.890555e+06 disk.sdb1.out‚0# proc.memory.rssMB‚ 3004.48046875 disk.dm-0.in‚ 1683.86328125 disk.sdb2.out‚ 986.740234375 disk.sda.out‚ 986.740234375 disk.sda1.in‚0 disk.sdb1.in‚0 disk.md0p1.out‚ 949.23046875 disk.nvme0n1.in‚ 939.5390625 disk.md0p1.in‚ 4.2890625) proc.memory.percent‚18.33789348602295 disk.rbd0.out‚ 2.466796875! disk.nvme1n1.out‚ 37627.328125 disk.sdb.in‚0.671875 disk.sda2.in‚ 3.6171875 disk.sda2.out‚ 986.740234375 disk.sda.in‚ 3.6171875 network.sent‚ 1.9519193e+07 disk.rbd2.out‚ 2.69140625 disk.sdb2.in‚0.671875 disk.rbd0.in‚0 cpu‚25.195913616129236) disk./.usagePercent‚75.80453399416217% disk./.usageGB‚2711.1373138427734" disk.nvme0n1.out‚ 96218.8203125 disk.md0.out‚ 949.23046875) proc.memory.availableMB‚ 1315.57421875 disk.dm-0.out‚133846.1484375 disk.rbd2.in‚0# memory_percent‚91.9703722000122 proc.cpu.threads‚17'OÀ}“Ï!‚(:ˆ çÕéÔÌ‘“m gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.099' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.27475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìîfÐ!‚(:Û çÕéÔŒçÅm disk.sda1.out‚0 disk.sdb.in‚0.671875# disk.nvme0n1.out‚96373.13671875 disk.sda.out‚ 988.822265625( proc.memory.availableMB‚ 1315.8046875 proc.cpu.threads‚17( disk./.usagePercent‚75.7897070069314 disk.md0p1.out‚ 951.2421875 disk.nvme0n1.in‚ 939.5390625 disk.md0.in‚ 4.2890625 disk.rbd2.out‚ 2.69140625 disk.sdb2.in‚0.671875 disk.rbd0.out‚ 2.466796875 disk.dm-0.out‚134000.7890625) proc.memory.percent‚18.33789348602295 disk.sda1.in‚0 disk.rbd2.in‚0$ memory_percent‚91.96896553039551 disk.sdb2.out‚ 988.822265625% disk./.usageGB‚2710.6070289611816 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.dm-0.in‚ 1683.86328125 disk.sda2.out‚ 988.822265625# proc.memory.rssMB‚ 3004.48046875 network.recv‚ 2.895599e+06 disk.md0p1.in‚ 4.2890625 disk.sdb.out‚ 988.822265625 disk.sda2.in‚ 3.6171875 disk.sda.in‚ 3.6171875 cpu‚25.195635034462175 network.sent‚ 1.962668e+07 disk.sdb1.out‚0 disk.nvme1n1.in‚ 744.32421875# disk.nvme1n1.out‚37627.65234375 disk.md0.out‚ 951.2421875g:4ëÑ!‚(:’ öÕéÔÚª©m gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚127.106' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚31.776499999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03 ‰‹cÒ!‚(:Ø öÕéÔâ™èm disk.nvme1n1.in‚ 744.32421875# disk.nvme1n1.out‚37627.83203125 disk.rbd0.in‚0 disk.dm-0.in‚ 1683.86328125 disk.sda1.in‚0 disk.md0p1.in‚ 4.2890625 network.sent‚ 1.9632958e+07 network.recv‚ 2.900339e+06$ disk./.usageGB‚2710.606960296631 disk.rbd2.in‚0 disk.sda.out‚990.0126953125# proc.memory.rssMB‚ 3004.48046875 cpu‚25.195332028595974 disk.rbd2.out‚ 2.69140625 disk.sdb2.in‚0.671875 disk.sdb2.out‚990.0126953125 disk.sda1.out‚0 disk.sdb1.out‚0 disk.md0p1.out‚952.375 disk.sda2.in‚ 3.6171875) proc.memory.availableMB‚ 1316.16015625! disk.dm-0.out‚134001.12890625 disk.sdb.out‚990.0126953125 disk.md0.out‚952.375 disk.sdb1.in‚0 disk.sda2.out‚990.0126953125 disk.sda.in‚ 3.6171875 disk.sdb.in‚0.671875 disk.nvme0n1.in‚ 939.5390625$ memory_percent‚91.96679592132568) proc.memory.percent‚18.33789348602295) disk./.usagePercent‚75.78970508704182 disk.md0.in‚ 4.2890625 disk.rbd0.out‚ 2.466796875! disk.nvme0n1.out‚ 96373.296875 proc.cpu.threads‚17¸#¥’Ó!‚(:‡ …ÖéÔ¬üÄm gpu.0.gpu‚4.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.051' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.01275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—ªspÔ!‚(:å …ÖéÔ‚íÞm disk.rbd0.out‚ 2.466796875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 1683.86328125# proc.memory.rssMB‚ 3004.48046875 disk.sda.out‚991.66943359375! disk.dm-0.out‚134369.16015625 disk.md0.in‚ 4.2890625% disk./.usageGB‚2710.6318588256836 disk.sdb.out‚991.66943359375 disk.nvme0n1.in‚ 939.5390625# disk.nvme0n1.out‚96740.91796875 cpu‚25.195060037273553 network.recv‚ 2.903607e+06) disk./.usagePercent‚75.79040126033499# memory_percent‚91.9668436050415) proc.memory.availableMB‚ 1316.15234375! disk.sdb2.out‚991.66943359375 disk.sdb.in‚0.671875 disk.sda2.in‚ 3.6171875 proc.cpu.threads‚17 disk.rbd2.out‚ 2.69140625 disk.rbd0.in‚0 disk.nvme1n1.in‚ 744.32421875" disk.nvme1n1.out‚ 37628.2421875 network.sent‚ 1.9640454e+07 disk.sdb2.in‚0.671875 disk.sda1.in‚0 disk.md0p1.out‚ 953.9765625 disk.md0.out‚ 953.9765625 disk.rbd2.in‚0 disk.sda.in‚ 3.6171875 disk.sda1.out‚0 disk.md0p1.in‚ 4.2890625! disk.sda2.out‚991.66943359375) proc.memory.percent‚18.33789348602295 †Ó2Õ!- _wandbruntime‚8905  _runtime‚8905¨SÕòÇÖ!Á trainer/global_step‚7549 % _timestamp‚1.7885048474927022e+09 _runtime‚8905.270099107 _step‚618 & lr-AdamW/pg1‚0.0003910564556043135 & lr-AdamW/pg2‚0.0003910564556043135 %M€Ì×!Æ _step‚618 & lr-AdamW/pg1‚0.0003910564556043135 & lr-AdamW/pg2‚0.0003910564556043135 trainer/global_step‚7549 % _timestamp‚1.7885048474927022e+09 _runtime‚8905.270099107êŽñÊ­2Ø!- _wandbruntime‚8905  _runtime‚8905d íAû Ù!öO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚94712236 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚645412659475456 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚702494194353664 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 191339349 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 349408071 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚636758577622528 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158068720 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 349381725 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚645413587449856 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 140574017 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚711533030855680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9039069478400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚645412659475456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45862406 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94712234 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚702494218293760 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158068722 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 158068722 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitÈõ;0or.on_train_batch_start/active.large_pool.freed‚ 158068720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚619 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _runtime‚8905.271742734 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191313005 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚636758559619584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9038836502016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8622652743526 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8654099855872 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 % _timestamp‚1.7885048474942026e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚645413587449856 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 191339349 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45861783 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 trainer/global_step‚7549 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚711533287772160 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8655009827328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640Ä?W?(Ú!ûO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚645413587449856 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 140574017 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚711533030855680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9039069478400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚645412659475456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45862406 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94712234 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚702494218293760 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158068722 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 158068722 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 158068720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚619 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _runtime‚8905.271742734 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191313005 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚636758559619584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9038836502016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8622652743526 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8654099855872 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 % _timestamp‚1.7885048474942026e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚645413587449856 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 191339349 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45861783 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 trainer/global_step‚7549 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚711533287772160 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8655009827328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚94712236 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚645412659475456 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚702494194353664 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 191339349 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 349408071 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚636758577622528 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158068720 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 349381725 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0늚óÑ2Û!- _wandbruntime‚8906  _runtime‚8906šZHè'Ü!€N VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚636840196708352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9040232087040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚615744775816480 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 349453088 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94724301 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚702584420058112 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94724299 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚615744758777120 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚624367620818920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚636840196708352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 349426716 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45867588 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158088882 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 191337836 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 191364206 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 191337836 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚702584443998208 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 191364206 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8655221316096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158088880 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚645495400021504 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 349426716 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140591887 _step‚620 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚645496328009216 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 158088880 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45868218 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0‡ª)£Ý!…N OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158088882 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 191337836 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 191364206 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 191337836 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚702584443998208 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 191364206 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8655221316096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158088880 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚645495400021504 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 349426716 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140591887 _step‚620 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚645496328009216 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 158088880 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45868218 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚636840196708352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9040232087040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚615744775816480 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 349453088 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94724301 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚702584420058112 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94724299 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚615744758777120 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚624367620818920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚636840196708352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 349426716 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45867588 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ìX2Þ!- _wandbruntime‚8906  _runtime‚8906‘?Þ_ß!Ù 0loss/FRt1/assignment_loss‚2.8191077296550455 /loss/FRt1/detection_loss‚1.9537231871874656 epoch‚6 % _timestamp‚1.7885048486650562e+09 /loss/FRt4/detection_loss‚3.5262949968639172 &loss/total_loss‚42.482525483706524 2loss/SRqqt1/assignment_loss‚1.6631002318771446 2loss/SRqqt2/assignment_loss‚1.1971088647842407 /loss/FBt1/detection_loss‚0.6961183191277087 /loss/FBt4/detection_loss‚2.1030035875737667 0loss/FRt3/assignment_loss‚1.3284357057677374 2loss/SRqqt3/assignment_loss‚1.0186422291090307 2loss/SRqqt4/assignment_loss‚1.7061305878491237 0loss/FBt2/assignment_loss‚0.1409974210466978 /loss/FRt2/detection_loss‚2.2648946838246453 .loss/FBt2/detection_loss‚2.715412676334381 _step‚621 1loss/FBt3/assignment_loss‚0.25516707834322006 1loss/SRqqt3/detection_loss‚0.9421173900203861 trainer/global_step‚7549 1loss/FBt4/assignment_loss‚0.43689668542312177 /loss/FBt3/detection_loss‚3.2786596417427063 0loss/FRt2/assignment_loss‚2.5188724676767986 0loss/SRqqt1/detection_loss‚1.767376208130051 0loss/SRqqt2/detection_loss‚2.385398576999533 1loss/SRqqt4/detection_loss‚2.4586380360455347 /loss/FRt4/assignment_loss‚3.632927019345133 1loss/FBt1/assignment_loss‚0.07992623938023939 /loss/FRt3/detection_loss‚1.5935759195988928 _runtime‚8906.4421528578:çdà!Þ /loss/FRt4/assignment_loss‚3.632927019345133 1loss/FBt1/assignment_loss‚0.07992623938023939 /loss/FRt3/detection_loss‚1.5935759195988928 _runtime‚8906.442152857 0loss/FRt1/assignment_loss‚2.8191077296550455 /loss/FRt1/detection_loss‚1.9537231871874656 epoch‚6 % _timestamp‚1.7885048486650562e+09 /loss/FRt4/detection_loss‚3.5262949968639172 &loss/total_loss‚42.482525483706524 2loss/SRqqt1/assignment_loss‚1.6631002318771446 2loss/SRqqt2/assignment_loss‚1.1971088647842407 /loss/FBt1/detection_loss‚0.6961183191277087 /loss/FBt4/detection_loss‚2.1030035875737667 0loss/FRt3/assignment_loss‚1.3284357057677374 2loss/SRqqt3/assignment_loss‚1.0186422291090307 2loss/SRqqt4/assignment_loss‚1.7061305878491237 0loss/FBt2/assignment_loss‚0.1409974210466978 /loss/FRt2/detection_loss‚2.2648946838246453 .loss/FBt2/detection_loss‚2.715412676334381 _step‚621 1loss/FBt3/assignment_loss‚0.25516707834322006 1loss/SRqqt3/detection_loss‚0.9421173900203861 trainer/global_step‚7549 1loss/FBt4/assignment_loss‚0.43689668542312177 /loss/FBt3/detection_loss‚3.2786596417427063 0loss/FRt2/assignment_loss‚2.5188724676767986 0loss/SRqqt1/detection_loss‚1.767376208130051 0loss/SRqqt2/detection_loss‚2.385398576999533 1loss/SRqqt4/detection_loss‚2.4586380360455347퀆¡ƒ“á!‚(:ˆ ”ÖéÔúšÌn gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚124.321' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.08025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~óH,câ!‚(:Ø ”ÖéÔõào) disk./.usagePercent‚75.79045917700392$ disk./.usageGB‚2710.633930206299$ memory_percent‚91.96977615356445 disk.sdb1.out‚0 network.sent‚ 1.9644396e+07 disk.nvme0n1.in‚ 939.5390625 disk.nvme1n1.in‚ 744.32421875 disk.md0.out‚ 955.8046875 disk.rbd2.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.md0p1.in‚ 4.2890625 disk.sda2.out‚ 993.568359375# disk.nvme1n1.out‚37705.38671875 disk.md0.in‚ 4.2890625' proc.memory.availableMB‚ 1315.671875# proc.memory.rssMB‚ 3004.48046875 disk.sdb2.in‚0.671875 disk.sda1.in‚0 disk.sda.in‚ 3.6171875 disk.sda.out‚ 993.568359375 disk.sdb.out‚ 993.568359375 network.recv‚ 2.908201e+06 disk.sdb1.in‚0 disk.md0p1.out‚ 955.8046875! disk.dm-0.out‚134655.94921875 disk.nvme0n1.out‚ 96950.5625 disk.rbd0.out‚ 2.466796875 disk.dm-0.in‚ 1683.86328125 disk.sdb2.out‚ 993.568359375 proc.cpu.threads‚17 disk.sdb.in‚0.671875 disk.sda2.in‚ 3.6171875 disk.rbd2.out‚ 2.69140625) proc.memory.percent‚18.33789348602295 cpu‚25.194780494464844úžàq“ã!‚(:ˆ £ÖéÔ¼Œm gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚131.173' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.79325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰AíÂeä!‚(:Ú £ÖéÔѦƒn disk.rbd0.in‚0$ memory_percent‚91.96844100952148 proc.cpu.threads‚17# disk.nvme0n1.out‚96951.05859375 network.sent‚ 1.9753922e+07) disk./.usagePercent‚75.79132995358816 disk.rbd2.in‚0 disk.rbd0.out‚ 2.466796875 disk.sdb1.in‚0! disk.sda2.out‚995.07763671875) proc.memory.percent‚18.33789348602295 disk.sdb.out‚995.07763671875' proc.memory.availableMB‚ 1315.890625 disk.sdb1.out‚0 disk.nvme0n1.in‚ 939.5390625" disk.nvme1n1.out‚ 37737.5859375 disk.sdb2.in‚0.671875 disk.sda.in‚ 3.6171875 disk.sda1.in‚0# proc.memory.rssMB‚ 3004.48046875 cpu‚25.194516268682733 disk.md0.out‚957.25 disk.rbd2.out‚ 2.69140625! disk.sdb2.out‚995.07763671875 disk.sda1.out‚0 disk.md0p1.in‚ 4.2890625 disk.dm-0.in‚ 1683.86328125% disk./.usageGB‚2710.6650733947754 disk.md0.in‚ 4.2890625 disk.md0p1.out‚957.25 disk.nvme1n1.in‚ 744.32421875 network.recv‚ 2.913313e+06 disk.sda.out‚995.07763671875! disk.dm-0.out‚134688.64453125 disk.sdb.in‚0.671875 disk.sda2.in‚ 3.6171875íÎÇv“å!‚(:ˆ ²ÖéÔëâŠm gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚125.603' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.40075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°Ô£ëgæ!‚(:Ü ²ÖéÔ×ðòm disk.rbd0.in‚0 disk.rbd0.out‚ 2.466796875 disk.dm-0.in‚ 1683.86328125 disk.sdb.in‚0.671875 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.dm-0.out‚ 134927.34375 disk.nvme1n1.in‚ 744.32421875 disk.md0.out‚ 958.55078125 disk.rbd2.out‚ 2.69140625$ memory_percent‚91.96991920471191 disk.sdb1.in‚0 network.sent‚ 1.9760234e+07) disk./.usagePercent‚75.79395870906284 disk.sdb2.in‚0.671875 disk.md0p1.out‚ 958.55078125# disk.nvme1n1.out‚37766.63671875 disk.rbd2.in‚0 network.recv‚ 2.917926e+06 disk.sda.out‚ 996.451171875 disk.nvme0n1.in‚ 939.5390625) proc.memory.percent‚18.33789348602295 disk.sda2.in‚ 3.6171875# disk.nvme0n1.out‚97160.70703125 disk.sda2.out‚ 996.451171875( proc.memory.availableMB‚ 1315.6484375 disk.sdb2.out‚ 996.451171875 disk.sda1.in‚0# proc.memory.rssMB‚ 3004.48046875 cpu‚25.194247381155364 disk.sda.in‚ 3.6171875 disk.sda1.out‚0 disk.md0p1.in‚ 4.2890625 disk.sdb.out‚ 996.451171875 disk.md0.in‚ 4.2890625$ disk./.usageGB‚2710.759090423584¶óä“ç!‚(:ˆ ÁÖéÔ´½âl gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚178.733' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.68325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½wÖOkè!‚(:à ÁÖéÔ£Á´n network.recv‚ 2.921661e+06 disk.sdb1.in‚0 disk.sdb.in‚0.671875 disk.sdb.out‚998.5654296875 disk.sda2.out‚998.5654296875$ memory_percent‚91.97149276733398 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sda2.in‚ 3.6171875 disk.sdb2.in‚0.671875 cpu‚25.193975550517546 disk.md0p1.in‚ 4.2890625 disk.md0.in‚ 4.2890625 disk.sdb2.out‚998.5654296875 disk.sda.in‚ 3.6171875' proc.memory.availableMB‚ 1315.390625 disk.sda1.out‚0 disk.nvme0n1.in‚ 939.5390625# disk.nvme0n1.out‚97160.79296875 disk.rbd2.out‚ 2.69140625 network.sent‚ 1.9765587e+07) disk./.usagePercent‚75.79746069431398 disk.dm-0.in‚ 1683.86328125 disk.nvme1n1.in‚ 744.32421875) proc.memory.percent‚18.33789348602295 disk.sda1.in‚0 disk.rbd0.in‚0 disk.md0p1.out‚ 960.5859375 disk.md0.out‚ 960.5859375 disk.sda.out‚998.5654296875% disk./.usageGB‚2710.8843383789062 disk.rbd0.out‚ 2.466796875 disk.sdb1.out‚0 disk.dm-0.out‚134985.3828125# disk.nvme1n1.out‚37824.58984375# proc.memory.rssMB‚ 3004.48046875û&2é!- _wandbruntime‚8963  _runtime‚8963T$Èê! trainer/global_step‚7599 $ _timestamp‚1.788504905485289e+09 _runtime‚8963.262513561 _step‚622 ' lr-AdamW/pg1‚0.00038372955438962755 ' lr-AdamW/pg2‚0.00038372955438962755JË–4Íë!Ç ' lr-AdamW/pg1‚0.00038372955438962755 ' lr-AdamW/pg2‚0.00038372955438962755 trainer/global_step‚7599 $ _timestamp‚1.788504905485289e+09 _runtime‚8963.262513561 _step‚622îg¿¯B2ì!- _wandbruntime‚8963  _runtime‚8963DÚ§Nü'í!öO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚640839531913728 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95315486 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 351634627 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8710187280896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚707005479576064 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9097222451712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚649549701191680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8710187280896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 192584251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 trainer/global_step‚7599 _runtime‚8963.263653856 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚640839513910784 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚649549701191680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9097453330944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 351660973 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚606 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚716102702027776 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 159076722 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚640839513910784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚628295167947770 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7885049054863436e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8711097252352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 159076720 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚619616620909776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚649550629166080 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 192557907 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46148788 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8678529998634 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 159076722 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8711097252352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328”’œõJ"î!ûO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚640839531913728 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95315486 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 351634627 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8710187280896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚707005479576064 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9097222451712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚649549701191680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8710187280896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 192584251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 trainer/global_step‚7599 _runtime‚8963.263653856 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚640839513910784 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚649549701191680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9097453330944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 351660973 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚606 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚716102702027776 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 159076722 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚640839513910784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚628295167947770 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7885049054863436e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8711097252352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 159076720 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚619616620909776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚649550629166080 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 192557907 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46148788 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8678529998634 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 159076722 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8711097252352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2ï#ûw2ï!- _wandbruntime‚8964  _runtime‚8964˾f'ð!€N OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚707095729220608 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8679647272120 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚649632441737728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220380160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141481508 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 351679618 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 192609108 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 159096880 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 351705990 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚624 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚640921132996608 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8678739296908 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚619695638504320 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8712218725888 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46153959 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚628374377801228 trainer/global_step‚7599 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 159096880 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 _runtime‚8964.359872562 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 351679618 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 192609108 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚640921150999552 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9098410395136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159096882 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 141482136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚619695655543680 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8711308741120 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚628375302815800 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 192582738 % _timestamp‚1.7885049065820646e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95327551 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 351705990 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚649633369725440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640ð‹ùÉ 'ñ!…N WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8679647272120 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚649632441737728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220380160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141481508 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 351679618 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 192609108 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 159096880 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 351705990 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚624 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚640921132996608 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8678739296908 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚619695638504320 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8712218725888 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46153959 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚628374377801228 trainer/global_step‚7599 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 159096880 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 _runtime‚8964.359872562 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 351679618 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 192609108 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚640921150999552 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9098410395136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159096882 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 141482136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚619695655543680 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8711308741120 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚628375302815800 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 192582738 % _timestamp‚1.7885049065820646e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95327551 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 351705990 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚649633369725440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚707095729220608ðÜÓå‘2ò!- _wandbruntime‚8964  _runtime‚8964ÁŸÖ$Yó!Ó /loss/FBt1/detection_loss‚0.7299248537243302 _runtime‚8964.360785946 0loss/FRt3/assignment_loss‚1.1993810093921164 2loss/SRqqt3/assignment_loss‚0.9814379319251026 1loss/FBt1/assignment_loss‚0.18881305993791916 1loss/FBt2/assignment_loss‚0.14836374975448052 1loss/SRqqt2/detection_loss‚2.3783834651112556 .loss/FRt4/detection_loss‚3.373058500289917 1loss/SRqqt3/detection_loss‚0.8435567062077743 .loss/FBt4/detection_loss‚2.359747269323894 % _timestamp‚1.7885049065837522e+09 1loss/FBt4/assignment_loss‚0.20919995906850936 /loss/FRt2/detection_loss‚2.2447706120354787 0loss/FBt3/assignment_loss‚0.6785822578822263 %loss/total_loss‚42.29361557955525 trainer/global_step‚7599 epoch‚6 1loss/SRqqt1/assignment_loss‚1.336219378809134 2loss/SRqqt4/assignment_loss‚1.4973372574205752 .loss/FBt3/detection_loss‚2.482777640223503 0loss/FRt1/assignment_loss‚2.5588709451258183 0loss/FRt4/assignment_loss‚3.2690523958206175 1loss/SRqqt2/assignment_loss‚2.249302413314581 0loss/SRqqt4/detection_loss‚2.474851398556321 .loss/FBt2/detection_loss‚2.605790227651596 /loss/FRt2/assignment_loss‚3.368359689809838 /loss/FRt3/detection_loss‚1.5940739149632661 _step‚625 /loss/FRt1/detection_loss‚1.4370891100891943 0loss/SRqqt1/detection_loss‚2.084671833117803÷%+^ô!Ø 0loss/FRt1/assignment_loss‚2.5588709451258183 0loss/FRt4/assignment_loss‚3.2690523958206175 1loss/SRqqt2/assignment_loss‚2.249302413314581 0loss/SRqqt4/detection_loss‚2.474851398556321 .loss/FBt2/detection_loss‚2.605790227651596 /loss/FRt2/assignment_loss‚3.368359689809838 /loss/FRt3/detection_loss‚1.5940739149632661 _step‚625 /loss/FRt1/detection_loss‚1.4370891100891943 0loss/SRqqt1/detection_loss‚2.084671833117803 /loss/FBt1/detection_loss‚0.7299248537243302 _runtime‚8964.360785946 0loss/FRt3/assignment_loss‚1.1993810093921164 2loss/SRqqt3/assignment_loss‚0.9814379319251026 1loss/FBt1/assignment_loss‚0.18881305993791916 1loss/FBt2/assignment_loss‚0.14836374975448052 1loss/SRqqt2/detection_loss‚2.3783834651112556 .loss/FRt4/detection_loss‚3.373058500289917 1loss/SRqqt3/detection_loss‚0.8435567062077743 .loss/FBt4/detection_loss‚2.359747269323894 % _timestamp‚1.7885049065837522e+09 1loss/FBt4/assignment_loss‚0.20919995906850936 /loss/FRt2/detection_loss‚2.2447706120354787 0loss/FBt3/assignment_loss‚0.6785822578822263 %loss/total_loss‚42.29361557955525 trainer/global_step‚7599 epoch‚6 1loss/SRqqt1/assignment_loss‚1.336219378809134 2loss/SRqqt4/assignment_loss‚1.4973372574205752 .loss/FBt3/detection_loss‚2.482777640223503ñW‘[rõ!‚(:ç ÐÖéÔ“•çm! disk.nvme1n1.out‚ 37920.765625 disk.sda.in‚ 3.6171875 proc.cpu.threads‚17 disk.md0p1.out‚ 962.16796875! disk.sda2.out‚1000.2080078125 disk.rbd2.in‚0 disk.sdb.in‚0.671875# disk.nvme0n1.out‚97527.66796875 disk.md0.out‚ 962.16796875$ memory_percent‚91.96865558624268# proc.memory.rssMB‚ 3004.48046875 network.recv‚ 2.926147e+06 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 2.466796875 disk.sdb1.out‚0! disk.dm-0.out‚135448.43359375 disk.sda.out‚1000.2080078125 cpu‚25.193683441899665 disk.dm-0.in‚ 1683.86328125 disk.sdb.out‚1000.2080078125 disk.nvme0n1.in‚ 939.5390625) proc.memory.percent‚18.33789348602295 disk.sda1.out‚0 disk.sdb1.in‚0 disk.md0p1.in‚ 4.2890625) proc.memory.availableMB‚ 1315.85546875 disk.nvme1n1.in‚ 744.32421875 disk.md0.in‚ 4.2890625 disk.rbd2.out‚ 2.69140625! disk.sdb2.out‚1000.2080078125 network.sent‚ 1.9782266e+07) disk./.usagePercent‚75.80443639977528% disk./.usageGB‚2711.1338233947754 disk.sda2.in‚ 3.6171875 disk.sdb2.in‚0.671875ÑÜT “ö!‚(:ˆ ÐÖéÔ†«ém gpu.0.gpu‚50.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚168.545' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.13625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Âì“÷!‚(:ˆ ßÖéÔŠÁÆm gpu.0.gpu‚51.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.235' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.80875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DÏyvø!‚(:ë ßÖéÔ¨·âm# disk.nvme1n1.out‚37998.25390625 disk.rbd2.out‚ 2.69140625 disk.md0p1.out‚ 963.56640625$ memory_percent‚91.96877479553223 network.recv‚ 2.931485e+06 disk.rbd2.in‚0! disk.sda.out‚1001.65966796875# disk.nvme0n1.out‚97527.78515625 disk.sda1.in‚0# proc.memory.rssMB‚ 3004.48046875 proc.cpu.threads‚17 disk.md0.in‚ 4.2890625 disk.md0.out‚ 963.56640625 network.sent‚ 1.9879253e+07! disk.sdb.out‚1001.65966796875 disk.nvme1n1.in‚ 744.32421875 disk.rbd0.out‚ 2.466796875% disk./.usageGB‚2711.1338233947754 disk.rbd0.in‚0) proc.memory.percent‚18.33789348602295) disk./.usagePercent‚75.80443639977528" disk.sda2.out‚1001.65966796875 disk.nvme0n1.in‚ 939.5390625" disk.sdb2.out‚1001.65966796875 cpu‚25.193416724883658 disk.sdb.in‚0.671875 disk.sdb2.in‚0.671875 disk.sda.in‚ 3.6171875 disk.sda1.out‚0 disk.sdb1.in‚0( proc.memory.availableMB‚ 1315.8359375 disk.sda2.in‚ 3.6171875 disk.sdb1.out‚0 disk.md0p1.in‚ 4.2890625 disk.dm-0.in‚ 1683.86328125 disk.dm-0.out‚135526.0390625!ÝÞ$’ù!‚(:‡ îÖéÔØm gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚129.282' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.3205 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôÍÒoú!‚(:ä îÖéԲņn) disk./.usagePercent‚75.80443639977528 disk.nvme0n1.in‚ 939.5390625% disk./.usageGB‚2711.1338233947754! disk.sda.out‚1003.05224609375 disk.md0p1.out‚ 964.89453125 disk.md0.in‚ 4.2890625 disk.rbd2.out‚ 2.69140625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.dm-0.in‚ 1683.86328125! disk.sdb.out‚1003.05224609375" disk.sda2.out‚1003.05224609375" disk.sdb2.out‚1003.05224609375 network.sent‚ 1.9883324e+07 disk.dm-0.out‚ 135526.25 disk.nvme1n1.in‚ 744.32421875 disk.md0.out‚ 964.89453125) proc.memory.availableMB‚ 1315.80859375) proc.memory.percent‚18.33789348602295 disk.sda.in‚ 3.6171875 disk.sda2.in‚ 3.6171875 disk.sdb2.in‚0.671875 disk.sda1.out‚0 disk.md0p1.in‚ 4.2890625 disk.rbd0.out‚ 2.466796875 disk.sdb.in‚0.671875" disk.nvme0n1.out‚ 97527.8046875 disk.rbd2.in‚0# memory_percent‚91.9689416885376# proc.memory.rssMB‚ 3004.48046875 cpu‚25.193121304824604 network.recv‚ 2.936099e+06 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0" disk.nvme1n1.out‚ 37998.4453125œo½Z“û!‚(:ˆ ýÖéÔË׉m gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚176.461' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.11525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S:Ækü!‚(:à ýÖéÔ›–n disk.sda1.in‚0) disk./.usagePercent‚75.80443639977528 disk.dm-0.out‚ 136037.75 disk.sda2.in‚ 3.6171875 disk.nvme1n1.in‚ 744.32421875 disk.rbd2.in‚0 cpu‚25.192883108105775 network.recv‚ 2.93965e+06 disk.sdb.in‚0.671875! disk.sda.out‚1004.96435546875 disk.rbd0.in‚0! disk.nvme1n1.out‚ 38143.015625 disk.md0.in‚ 4.2890625 disk.md0.out‚ 966.73828125" disk.sdb2.out‚1004.96435546875 disk.dm-0.in‚ 1683.86328125 network.sent‚ 1.9890943e+07% disk./.usageGB‚2711.1338233947754! disk.sdb.out‚1004.96435546875 disk.nvme0n1.in‚ 939.5390625! disk.nvme0n1.out‚ 97894.734375' proc.memory.availableMB‚ 1315.796875 disk.md0p1.out‚ 966.73828125 disk.rbd2.out‚ 2.69140625 disk.sdb2.in‚0.671875 disk.sda.in‚ 3.6171875 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd0.out‚ 2.466796875# proc.memory.rssMB‚ 3004.48046875) proc.memory.percent‚18.33789348602295 disk.md0p1.in‚ 4.2890625" disk.sda2.out‚1004.96435546875 disk.sda1.out‚0$ memory_percent‚91.969013214111338e—2ý!- _wandbruntime‚9020  _runtime‚9020Úµ¸áÇþ!Á & lr-AdamW/pg1‚0.0003764323389815278 & lr-AdamW/pg2‚0.0003764323389815278 trainer/global_step‚7649 % _timestamp‚1.7885049629203234e+09 _runtime‚9020.697563988 _step‚6269fâ¼Ìÿ!Æ trainer/global_step‚7649 % _timestamp‚1.7885049629203234e+09 _runtime‚9020.697563988 _step‚626 & lr-AdamW/pg1‚0.0003764323389815278 & lr-AdamW/pg2‚0.0003764323389815278ò/½KÒ2€"- _wandbruntime‚9020  _runtime‚9020ƒróÛ"õO JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚653686742907904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 160084720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚644920468201984 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 160084722 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚720672162383360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚644920486204928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8767184677376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚627 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 193802809 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 353887529 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚653686742907904 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 160084720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 $ _timestamp‚1.788504962921449e+09 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 142353764 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346¶@`¤(‚"úO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚644920468201984 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 160084722 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚720672162383360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚644920486204928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8767184677376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚627 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 193802809 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 353887529 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚653686742907904 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 160084720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 $ _timestamp‚1.788504962921449e+09 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 142353764 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚653686742907904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 160084720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944ó‹Á²Š2ƒ"- _wandbruntime‚9021  _runtime‚9021¨!Ç'„"€N >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8735524527228 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95930799 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚623646535270880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 % _timestamp‚1.7885049640757568e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8767396166144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚606 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8734616552016 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 353958892 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 160104880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 353958892 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8768306150912 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 193827640 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚632382059798108 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 142372125 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9156561900544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 193854010 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚7649 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚653770411441664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚645002087287808 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142371519 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚720763552403456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚604 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 193827640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚645002105290752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _runtime‚9021.853777788 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46440720 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 193854010 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46441324 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚711607014443008 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615÷8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8735524527228 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95930799 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚623646535270880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 % _timestamp‚1.7885049640757568e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8767396166144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚606 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8734616552016 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitI«Ôor.on_train_batch_end/allocation.all.allocated‚ 353958892 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 160104880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 353958892 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8768306150912 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 193827640 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚632382059798108 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 142372125 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9156561900544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 193854010 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚7649 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚653770411441664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚645002087287808 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142371519 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚720763552403456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚604 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 193827640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚645002105290752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _runtime‚9021.853777788 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46440720 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 193854010 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46441324ôXU2†"- _wandbruntime‚9021  _runtime‚9021q1â[‡"Õ 0loss/FRt3/assignment_loss‚1.6381644819464003 1loss/SRqqt4/assignment_loss‚1.375591258807429 1loss/SRqqt1/detection_loss‚1.7695377484346047 0loss/SRqqt2/detection_loss‚2.802546524483225 1loss/SRqqt3/detection_loss‚0.9739009120302009 $ _timestamp‚1.788504964077675e+09 .loss/FRt1/detection_loss‚2.101516574396454 .loss/FRt2/detection_loss‚2.517009853571653 0loss/FRt1/assignment_loss‚2.1894751217630173 0loss/FRt2/assignment_loss‚3.3078978598117827 /loss/FRt4/detection_loss‚3.4442601908336985 2loss/SRqqt3/assignment_loss‚0.9938191634552952 .loss/FRt3/detection_loss‚1.645649482895221 /loss/FBt2/detection_loss‚2.4084204855290325 /loss/FBt3/detection_loss‚2.6470992234018116 epoch‚6 /loss/FRt4/assignment_loss‚2.750619126991792 1loss/SRqqt4/detection_loss‚2.3057617668447823 %loss/total_loss‚41.25354152304852 _step‚629 2loss/SRqqt1/assignment_loss‚1.5503037667427308 1loss/FBt2/assignment_loss‚0.21298602749795298 1loss/FBt3/assignment_loss‚0.15927690966053534 _runtime‚9021.854712892 0loss/FBt4/assignment_loss‚0.2406226679341719 /loss/FBt1/detection_loss‚0.5449641341609615 /loss/FBt4/detection_loss‚1.7575917447868146 trainer/global_step‚7649 2loss/SRqqt2/assignment_loss‚1.7894189616908198 1loss/FBt1/assignment_loss‚0.12710753537812824ˆ•Œ`ˆ"Ú 1loss/SRqqt1/detection_loss‚1.7695377484346047 0loss/SRqqt2/detection_loss‚2.802546524483225 1loss/SRqqt3/detection_loss‚0.9739009120302009 $ _timestamp‚1.788504964077675e+09 .loss/FRt1/detection_loss‚2.101516574396454 .loss/FRt2/detection_loss‚2.517009853571653 0loss/FRt1/assignment_loss‚2.1894751217630173 0loss/FRt2/assignment_loss‚3.3078978598117827 /loss/FRt4/detection_loss‚3.4442601908336985 2loss/SRqqt3/assignment_loss‚0.9938191634552952 .loss/FRt3/detection_loss‚1.645649482895221 /loss/FBt2/detection_loss‚2.4084204855290325 /loss/FBt3/detection_loss‚2.6470992234018116 epoch‚6 /loss/FRt4/assignment_loss‚2.750619126991792 1loss/SRqqt4/detection_loss‚2.3057617668447823 %loss/total_loss‚41.25354152304852 _step‚629 2loss/SRqqt1/assignment_loss‚1.5503037667427308 1loss/FBt2/assignment_loss‚0.21298602749795298 1loss/FBt3/assignment_loss‚0.15927690966053534 _runtime‚9021.854712892 0loss/FBt4/assignment_loss‚0.2406226679341719 /loss/FBt1/detection_loss‚0.5449641341609615 /loss/FBt4/detection_loss‚1.7575917447868146 trainer/global_step‚7649 2loss/SRqqt2/assignment_loss‚1.7894189616908198 1loss/FBt1/assignment_loss‚0.12710753537812824 0loss/FRt3/assignment_loss‚1.6381644819464003 1loss/SRqqt4/assignment_loss‚1.375591258807429õ³c“‰"‚(:ˆ Œ×éÔãÙ¬m gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚124.321' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.08025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ý°MtlŠ"‚(:á Œ×éÔ׈ýn disk.sdb.in‚3.78125! disk.sdb2.out‚1006.8974609375 disk.dm-0.in‚ 1683.86328125 disk.nvme0n1.in‚ 939.5390625! disk.nvme0n1.out‚ 97894.734375$ memory_percent‚91.96927547454834) proc.memory.percent‚18.33789348602295 proc.cpu.threads‚17 disk.sdb.out‚1006.8974609375! disk.sda2.out‚1006.8974609375 disk.rbd2.in‚0 disk.rbd2.out‚ 2.69140625 disk.sda.in‚8.515625 disk.sdb1.in‚0 disk.sdb2.in‚3.78125 disk.sda1.in‚0 disk.rbd0.out‚ 2.466796875 cpu‚25.192586533374747) disk./.usagePercent‚75.80443639977528% disk./.usageGB‚2711.1338233947754 disk.sdb1.out‚0" disk.nvme1n1.out‚ 38143.2109375 disk.md0.out‚ 968.60546875 disk.rbd0.in‚0 network.recv‚ 2.944004e+06 disk.md0p1.in‚ 12.296875 disk.dm-0.out‚136037.9453125 disk.md0.in‚ 12.296875) proc.memory.availableMB‚ 1315.75390625# proc.memory.rssMB‚ 3004.48046875 disk.sda2.in‚8.515625 disk.nvme1n1.in‚ 744.32421875 disk.sda.out‚1006.8974609375 disk.sda1.out‚0 network.sent‚ 1.9907514e+07 disk.md0p1.out‚ 968.60546875Ȭeà’‹"‚(:‡ ›×éÔõ±¸m gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚122.486' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.6215 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07§jŒ"‚(:ß ›×éÔ·§Òm$ memory_percent‚91.96939468383789# proc.memory.rssMB‚ 3004.48046875) proc.memory.percent‚18.33789348602295 disk.sdb.in‚3.78125 disk.sdb2.in‚3.78125 disk.sda1.in‚0 disk.sda.out‚1008.4169921875 cpu‚25.192303368848236 disk.md0.in‚ 12.296875 disk.rbd2.in‚0 disk.sda2.in‚8.515625' proc.memory.availableMB‚ 1315.734375 proc.cpu.threads‚17 disk.md0p1.in‚ 12.296875 disk.md0p1.out‚ 970.0546875" disk.nvme0n1.out‚ 98261.8203125 disk.md0.out‚ 970.0546875 disk.sda.in‚8.515625 disk.sda1.out‚0 disk.sdb1.in‚0 disk.sdb.out‚1008.4169921875 disk.nvme0n1.in‚ 939.5390625 disk.rbd0.in‚0 disk.sdb1.out‚0 network.sent‚ 2.0004436e+07( disk./.usagePercent‚75.7896923944385% disk./.usageGB‚2710.6065063476562 disk.dm-0.in‚ 1683.86328125! disk.dm-0.out‚136549.63671875 disk.nvme1n1.in‚ 744.32421875! disk.sdb2.out‚1008.4169921875 disk.rbd0.out‚ 2.466796875 network.recv‚ 2.949362e+06! disk.sda2.out‚1008.4169921875# disk.nvme1n1.out‚38287.81640625 disk.rbd2.out‚ 2.69140625i"»l"‚(:á ª×éÔÙ·m disk.nvme0n1.in‚ 939.5390625" disk.nvme0n1.out‚ 98261.8203125 network.sent‚ 2.0008553e+07) disk./.usagePercent‚75.78969346104381 disk.rbd0.in‚0 disk.md0p1.in‚ 12.296875! disk.sdb.out‚1010.28564453125 disk.sda2.in‚8.515625" disk.sda2.out‚1010.28564453125 disk.nvme1n1.in‚ 744.32421875 network.recv‚ 2.954221e+06 disk.sdb1.in‚0 disk.nvme1n1.out‚ 38287.96875! disk.sda.out‚1010.28564453125 disk.rbd0.out‚ 2.466796875 disk.dm-0.in‚ 1683.86328125# proc.memory.rssMB‚ 3004.48046875) proc.memory.percent‚18.33789348602295$ disk./.usageGB‚2710.606544494629 disk.sdb1.out‚0 disk.sdb.in‚3.78125 disk.md0.in‚ 12.296875 disk.rbd2.in‚0 disk.rbd2.out‚ 2.69140625" disk.sdb2.out‚1010.28564453125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.dm-0.out‚136549.7890625 disk.sdb2.in‚3.78125 disk.sda.in‚8.515625 disk.md0p1.out‚ 971.85546875# memory_percent‚91.9724702835083) proc.memory.availableMB‚ 1315.23046875 disk.sda1.out‚0 disk.md0.out‚ 971.85546875 cpu‚25.19206969948107”CRp’Ž"‚(:‡ ª×éÔÜáªm gpu.0.gpu‚27.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚181.414' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.3535 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýa%—’"‚(:‡ ¹×éÔ¶á™m gpu.0.gpu‚27.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚147.606' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.9015 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U8nd"‚(:Ù ¹×éÔ†ÏÏm disk.rbd0.out‚ 2.466796875 disk.sdb1.in‚0 disk.sda1.in‚0 disk.dm-0.out‚ 136551.359375 disk.sdb.in‚3.78125) proc.memory.percent‚18.33789348602295 disk.rbd2.out‚ 2.69140625 disk.sda.in‚8.515625 disk.md0p1.out‚ 973.2734375 disk.sdb.out‚1011.759765625 disk.nvme0n1.in‚ 939.5390625 cpu‚25.19180691865547 proc.cpu.threads‚17# disk.nvme0n1.out‚98262.05859375' proc.memory.availableMB‚ 1315.609375# proc.memory.rssMB‚ 3004.48046875 network.recv‚ 2.956903e+06 disk.sdb2.out‚1011.759765625 disk.sdb1.out‚0$ memory_percent‚91.97015762329102 network.sent‚ 2.001289e+07) disk./.usagePercent‚75.78829108836749% disk./.usageGB‚2710.5563888549805 disk.rbd0.in‚0 disk.md0p1.in‚ 12.296875 disk.dm-0.in‚ 1683.86328125 disk.nvme1n1.in‚ 744.32421875# disk.nvme1n1.out‚38289.30078125 disk.md0.out‚ 973.2734375 disk.rbd2.in‚0 disk.sda.out‚1011.759765625 disk.sda2.in‚8.515625 disk.sdb2.in‚3.78125 disk.sda1.out‚0 disk.sda2.out‚1011.759765625 disk.md0.in‚ 12.296875ØJ2‘"- _wandbruntime‚9077  _runtime‚9077¾qÖÉ’"à trainer/global_step‚7699 % _timestamp‚1.7885050202021084e+09 _runtime‚9077.979337173 _step‚630 ' lr-AdamW/pg1‚0.00036916633010876843 ' lr-AdamW/pg2‚0.00036916633010876843DÔæQΓ"È ' lr-AdamW/pg1‚0.00036916633010876843 ' lr-AdamW/pg2‚0.00036916633010876843 trainer/global_step‚7699 % _timestamp‚1.7885050202021084e+09 _runtime‚9077.979337173 _step‚630ö—s•2”"- _wandbruntime‚9077  _runtime‚9077š•=ìü'•"öO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143245313 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚7699 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8823272102400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚649001440496128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚649001422493184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 161092720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚96521984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚636308681912386 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚627518397403536 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8822362130944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚649001422493184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8823272102400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 161092720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46723327 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9213779260928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚657824712598528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143244676 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9214003848704 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8790284508850 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46722692 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚657823784624128 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 195047711 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 356166777 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 195047711 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚657823784624128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 195074055 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚716028073960960 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚627518380364176 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚657824712598528 _runtime‚9077.980402348 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8822362130944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8789376533846 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚649001440496128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0‰öx#Û–"ûO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚649001422493184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8823272102400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 161092720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46723327 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9213779260928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚657824712598528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143244676 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9214003848704 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8790284508850 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46722692 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚657823784624128 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 195047711 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 356166777 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 195047711 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚657823784624128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 195074055 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚716028073960960 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚627518380364176 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚657824712598528 _runtime‚9077.980402348 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8822362130944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8789376533846 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚649001440496128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143245313 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚7699 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8823272102400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚649001440496128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚649001422493184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 J<©? ×& DeviceStatsMonitor.on_train_batch_start/allocation.all.freed‚ 356140431 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚631 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 161092720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚96521984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚636308681912386 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚627518397403536 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8822362130944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521÷™†ª!2—"- _wandbruntime‚9079  _runtime‚9079ûœø'˜"€N FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143262353 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚657907453157888 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 356185422 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚7699 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚657907453157888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96534049 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9214922748416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚716118299665408 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 356185422 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 271583232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 % _timestamp‚1.7885050213502388e+09 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 247643136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚627597397958720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143262992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 195072542 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 356211794 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8824393575936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8823483591168 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 161112880 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8823483591168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46728304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 _step‚632 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚725333470056960 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚636388816780416 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚649083059581952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚716118275725312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚649083059581952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46728941 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8790493807124 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚649083041579008 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0õ×1ç '™"…N B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 195072542 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 356211794 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8824393575936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8823483591168 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 161112880 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8823483591168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46728304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 _step‚632 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚725333470056960 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚636388816780416 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚649083059581952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚716118275725312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚649083059581952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46728941 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8790493807124 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚649083041579008 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143262353 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚657907453157888 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 356185422 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚7699 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚657907453157888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96534049 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9214922748416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚716118299665408 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 356185422 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 271583232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 % _timestamp‚1.7885050213502388e+09 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 247643136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚627597397958720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143262992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144øræ_2š"- _wandbruntime‚9079  _runtime‚9079poe2V›"Ð /loss/SRqqt4/detection_loss‚3.03106842611147 /loss/FRt3/assignment_loss‚2.130174965597689 1loss/FBt2/assignment_loss‚0.03763526743193779 1loss/SRqqt1/detection_loss‚2.0858931595087054 /loss/FRt1/detection_loss‚1.8153475379574957 .loss/FBt2/detection_loss‚2.167544849216938 2loss/SRqqt4/assignment_loss‚1.6343597973818365 1loss/FBt4/assignment_loss‚0.27624096409621973 trainer/global_step‚7699 .loss/FRt4/assignment_loss‚3.33968456586202 /loss/FBt1/detection_loss‚0.6381579749827642 epoch‚6 0loss/SRqqt2/detection_loss‚2.537778463628557 /loss/FBt4/detection_loss‚2.1646773035709677 %loss/total_loss‚42.47489743203768 /loss/FRt2/assignment_loss‚3.016714965519698 2loss/SRqqt1/assignment_loss‚1.8934706342220307 1loss/FBt3/assignment_loss‚0.03502785865415111 /loss/FRt3/detection_loss‚1.7434375573357102 -loss/FRt4/detection_loss‚3.32581078261137 0loss/FRt1/assignment_loss‚2.0152582213282586 2loss/SRqqt3/assignment_loss‚1.1506903195358087 1loss/FBt1/assignment_loss‚0.19645782184368615 % _timestamp‚1.7885050213522112e+09 _runtime‚9079.129234398 _step‚633 0loss/SRqqt2/assignment_loss‚1.78274162168856 .loss/FBt3/detection_loss‚2.215604497326745 /loss/FRt2/detection_loss‚2.3692553785184156 1loss/SRqqt3/detection_loss‚0.8718644981066473cûo[œ"Õ .loss/FBt3/detection_loss‚2.215604497326745 /loss/FRt2/detection_loss‚2.3692553785184156 1loss/SRqqt3/detection_loss‚0.8718644981066473 /loss/SRqqt4/detection_loss‚3.03106842611147 /loss/FRt3/assignment_loss‚2.130174965597689 1loss/FBt2/assignment_loss‚0.03763526743193779 1loss/SRqqt1/detection_loss‚2.0858931595087054 /loss/FRt1/detection_loss‚1.8153475379574957 .loss/FBt2/detection_loss‚2.167544849216938 2loss/SRqqt4/assignment_loss‚1.6343597973818365 1loss/FBt4/assignment_loss‚0.27624096409621973 trainer/global_step‚7699 .loss/FRt4/assignment_loss‚3.33968456586202 /loss/FBt1/detection_loss‚0.6381579749827642 epoch‚6 0loss/SRqqt2/detection_loss‚2.537778463628557 /loss/FBt4/detection_loss‚2.1646773035709677 %loss/total_loss‚42.47489743203768 /loss/FRt2/assignment_loss‚3.016714965519698 2loss/SRqqt1/assignment_loss‚1.8934706342220307 1loss/FBt3/assignment_loss‚0.03502785865415111 /loss/FRt3/detection_loss‚1.7434375573357102 -loss/FRt4/detection_loss‚3.32581078261137 0loss/FRt1/assignment_loss‚2.0152582213282586 2loss/SRqqt3/assignment_loss‚1.1506903195358087 1loss/FBt1/assignment_loss‚0.19645782184368615 % _timestamp‚1.7885050213522112e+09 _runtime‚9079.129234398 _step‚633 0loss/SRqqt2/assignment_loss‚1.78274162168856ùNl“"‚(:ˆ È×éÔæÇn gpu.0.gpu‚40.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü Kœhž"‚(:Ý È×éÔ‹ÈÇn! disk.sda2.out‚1013.2783203125 disk.nvme0n1.in‚ 939.5390625$ memory_percent‚91.97089672088623 disk.rbd0.in‚0 disk.md0.in‚ 12.296875 disk.sda1.out‚0 disk.sdb1.in‚0 disk.md0p1.in‚ 12.296875 disk.sdb.in‚3.78125) proc.memory.availableMB‚ 1315.48828125) proc.memory.percent‚18.33789348602295 network.sent‚ 2.0018934e+07 disk.sda1.in‚0 disk.rbd0.out‚ 2.466796875 disk.dm-0.out‚136995.4140625 disk.rbd2.out‚ 2.69140625 disk.sdb2.in‚3.78125 disk.nvme0n1.out‚ 98628.9375 disk.sda.in‚8.515625 disk.sdb1.out‚0 disk.md0p1.out‚ 974.7265625" disk.nvme1n1.out‚ 38366.4765625) disk./.usagePercent‚75.79039208752921 disk.sda.out‚1013.2783203125 cpu‚25.191540721399168 proc.cpu.threads‚17 disk.rbd2.in‚0! disk.sdb2.out‚1013.2783203125 disk.dm-0.in‚ 1683.86328125 disk.sdb.out‚1013.2783203125 disk.sda2.in‚8.515625 disk.md0.out‚ 974.7265625# proc.memory.rssMB‚ 3004.48046875 disk.nvme1n1.in‚ 744.32421875 network.recv‚ 2.961196e+06% disk./.usageGB‚2710.6315307617188aûÈ^lŸ"‚(:á ××éÔØȃo disk.md0.out‚ 976.6796875 disk.rbd0.in‚0 disk.sdb.out‚1015.2939453125 disk.sda2.in‚8.515625 disk.sda.in‚8.515625) proc.memory.availableMB‚ 1315.47265625 disk.sda1.out‚0$ memory_percent‚91.97099208831787# proc.memory.rssMB‚ 3004.48046875) proc.memory.percent‚18.33789348602295% disk./.usageGB‚2710.6652450561523# disk.nvme1n1.out‚38366.69921875 disk.rbd2.in‚0" disk.nvme0n1.out‚ 98629.1015625 cpu‚25.191278103480336) disk./.usagePercent‚75.79133475331211 disk.rbd0.out‚ 2.466796875 disk.nvme1n1.in‚ 744.32421875 disk.sdb2.in‚3.78125! disk.sdb2.out‚1015.2939453125 disk.sda.out‚1015.2939453125 disk.md0p1.in‚ 12.296875! disk.dm-0.out‚136995.80078125! disk.sda2.out‚1015.2939453125 network.sent‚ 2.0128707e+07 disk.md0.in‚ 12.296875 disk.rbd2.out‚ 2.69140625 disk.sda1.in‚0 disk.sdb1.in‚0 disk.dm-0.in‚ 1683.86328125 disk.nvme0n1.in‚ 939.5390625 network.recv‚ 2.96649e+06 disk.sdb1.out‚0 disk.md0p1.out‚ 976.6796875 disk.sdb.in‚3.78125 proc.cpu.threads‚17N36“ "‚(:ˆ ××éÔ—œ”o gpu.0.gpu‚54.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚143.263' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.81575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýÁ“¡"‚(:ˆ æ×éÔ‰åÉm gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚117.525' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.38125 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0È€îÍ[¢"‚(:Ð æ×éÔ…ä„n disk.rbd0.in‚0 disk.rbd0.out‚ 2.466796875 disk.md0p1.out‚ 978.03125 disk.dm-0.out‚ 137422.984375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.md0.out‚ 978.03125 network.sent‚ 2.0135028e+07 network.recv‚ 2.97132e+06 disk.sda2.in‚8.515625! disk.sda2.out‚1016.7080078125 disk.sda.in‚8.515625# memory_percent‚91.9701337814331# proc.memory.rssMB‚ 3004.48046875 disk.nvme0n1.in‚ 939.5390625! disk.nvme0n1.out‚ 98995.984375 disk.nvme1n1.in‚ 744.32421875) proc.memory.percent‚18.33789348602295 cpu‚25.191024729296863 disk.sdb.out‚1016.7080078125 disk.md0.in‚ 12.296875 disk.sda.out‚1016.7080078125 disk.sda1.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚ 12.296875 disk.dm-0.in‚ 1683.86328125% disk./.usageGB‚2710.7590408325195 disk.rbd2.in‚0! disk.sdb2.out‚1016.7080078125 disk.rbd2.out‚ 2.69140625) proc.memory.availableMB‚ 1315.61328125 disk.nvme1n1.out‚38427 disk.sdb2.in‚3.78125) disk./.usagePercent‚75.79395732247592 disk.sdb.in‚3.78125FÕi“£"‚(:ˆ õ×éÔ…ž—m gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.059' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.26475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZƒÉp¤"‚(:å õ×éÔï†n network.recv‚ 2.974771e+06! disk.sdb.out‚1018.16552734375# disk.nvme0n1.out‚98996.16015625 disk.sdb1.out‚0 cpu‚25.19073939386681 disk.sda1.out‚0 disk.sdb1.in‚0 network.sent‚ 2.0140272e+07) disk./.usagePercent‚75.79746485407473 disk.nvme1n1.in‚ 744.32421875 disk.sda.in‚8.515625 disk.rbd0.in‚0) proc.memory.percent‚18.33789348602295 proc.cpu.threads‚17 disk.sda2.in‚8.515625 disk.rbd2.in‚0 disk.sdb2.in‚3.78125! disk.sda.out‚1018.16552734375 disk.sda1.in‚0# proc.memory.rssMB‚ 3004.48046875 disk.sdb.in‚3.78125" disk.sda2.out‚1018.16552734375" disk.sdb2.out‚1018.16552734375! disk.dm-0.out‚137452.51171875 disk.nvme0n1.in‚ 939.5390625 disk.md0.in‚ 12.296875 disk.md0p1.out‚ 979.4453125) proc.memory.availableMB‚ 1315.44140625% disk./.usageGB‚2710.8844871520996" disk.nvme1n1.out‚ 38456.3515625 disk.md0p1.in‚ 12.296875 disk.dm-0.in‚ 1683.86328125$ memory_percent‚91.97118282318115 disk.md0.out‚ 979.4453125 disk.rbd2.out‚ 2.69140625 disk.rbd0.out‚ 2.466796875°%,2¥"- _wandbruntime‚9135  _runtime‚9135aÐy¢È¦" _runtime‚ 9135.83141583 _step‚634 ' lr-AdamW/pg1‚0.00036193304199670754 ' lr-AdamW/pg2‚0.00036193304199670754 trainer/global_step‚7749 % _timestamp‚1.7885050780540397e+09>§1]Q§"Ç % _timestamp‚1.7885050780540397e+09 _runtime‚ 9135.83141583 _stee·.)|p‚634 ' lr-AdamW/pg1‚0.00036193304199670754 ' lr-AdamW/pg2‚0.00036193304199670754 trainer/global_step‚7749ú2öiR2¨"- _wandbruntime‚9135  _runtime‚9135ð¨6žü'©"öO [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _step‚635 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8879359527424 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 358419679 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 162100722 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 196318957 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235944960 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 358393333 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144137616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚97125236 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47012380 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47011741 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 144136975 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 162100722 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 162100720 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 196292613 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8878449555968 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚631469277130736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97125234 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚729811819643392 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚661960826340352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8879359527424 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚640314513880330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚661961754314752 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚720539335243264 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212004864 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚631469260091376 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 % _timestamp‚1.7885050780557196e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚641 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚729811583698432 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9272460460032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9272248455168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚661960826340352 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚640315438894694 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚653082394787328 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8878449555968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8879359527424 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 358419679 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 162100722 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 196318957 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235944960 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 358393333 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144137616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚97125236 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47012380 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47011741 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 144136975 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 162100722 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 162100720 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 196292613 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8878449555968 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚631469277130736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97125234 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚729811819643392 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚661960826340352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8879359527424 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚640314513880330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚661961754314752 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚720539335243264 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212004864 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚631469260091376 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 % _timestamp‚1.7885050780557196e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚641 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚729811583698432 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9272460460032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9272248455168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚661960826340352 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚640315438894694 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚653082394787328 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8878449555968û6P.2«"- _wandbruntime‚9136  _runtime‚9136IË1¬'¬"þM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9273421731840 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8880481000960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 358464696 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚720629560947712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8847279037444 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8846371062232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚7749 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚720629584887808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _step‚636 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 196343814 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144154776 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8879571016192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97137301 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚640395573762724 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚653164013873152 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 162120880 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _runtime‚ 9136.93605734 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 358438324 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚653163995870208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47018095 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196343814 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 358464696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚631548277685920 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 162120880 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97137299 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 $ _timestamp‚1.788505079158237e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚662044494874112 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚662043566886400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚729902982679552 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8879571016192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚662044494874112 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 196317444 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚653164013873152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9273648403456 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚640394648748152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0È¢í¤î­"ƒN S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚662043566886400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚729902982679552 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8879571016192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚662044494874112 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 196317444 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚653164013873152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9273648403456 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.cu¯L²$rrent‚26370 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚640394648748152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9273421731840 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8880481000960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 358464696 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚720629560947712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8847279037444 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8846371062232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚7749 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚720629584887808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _step‚636 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 196343814 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144154776 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8879571016192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97137301 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚640395573762724 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚653164013873152 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 162120880 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _runtime‚ 9136.93605734 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 358438324 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚653163995870208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47018095 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196343814 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 358464696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚631548277685920 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 162120880 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97137299 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 $ _timestamp‚1.788505079158237e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚662044494874112 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768üªûñ2®"- _wandbruntime‚9136  _runtime‚9136ðŒõU¯"Ï .loss/FRt4/detection_loss‚2.833920208613078 .loss/FRt1/detection_loss‚1.543376781815273 /loss/FRt2/detection_loss‚2.1003872300295727 /loss/FBt3/detection_loss‚2.6848136305809023 %loss/total_loss‚40.03479764083455 $ _timestamp‚1.788505079160162e+09 0loss/FRt4/assignment_loss‚3.0526657660802203 /loss/FRt1/assignment_loss‚2.120944380760193 1loss/SRqqt1/assignment_loss‚1.271113364893044 1loss/SRqqt2/assignment_loss‚1.232062656552561 1loss/SRqqt3/detection_loss‚1.0862444825656712 /loss/FRt2/assignment_loss‚3.766853988170624 1loss/SRqqt4/assignment_loss‚1.419683252993439 0loss/SRqqt2/detection_loss‚2.335174060636951 epoch‚6 1loss/FBt1/assignment_loss‚0.12050787154074903 0loss/FBt2/assignment_loss‚0.2022906296760567 /loss/FRt3/detection_loss‚1.9185970960184933 0loss/SRqqt4/detection_loss‚2.519571304321289 .loss/FBt2/detection_loss‚2.242289464920759 1loss/FBt3/assignment_loss‚0.13575044660829008 0loss/SRqqt1/detection_loss‚1.951503809156089 _runtime‚9136.937150995 _step‚637 0loss/FRt3/assignment_loss‚1.5790430545806884 2loss/SRqqt3/assignment_loss‚0.9136479856050573 0loss/FBt4/assignment_loss‚0.3892026714590732 /loss/FBt1/detection_loss‚0.6570605832710177 /loss/FBt4/detection_loss‚1.9580929199854533 trainer/global_step‚7749ìÖoÿZ°"Ô /loss/FBt4/detection_loss‚1.9580929199854533 trainer/global_step‚7749 .loss/FRt4/detection_loss‚2.833920208613078 .loss/FRt1/detection_loss‚1.543376781815273 /loss/FRt2/detection_loss‚2.1003872300295727 /loss/FBt3/detection_loss‚2.6848136305809023 %loss/total_loss‚40.03479764083455 $ _timestamp‚1.788505079160162e+09 0loss/FRt4/assignment_loss‚3.0526657660802203 /loss/FRt1/assignment_loss‚2.120944380760193 1loss/SRqqt1/assignment_loss‚1.271113364893044 1loss/SRqqt2/assignment_loss‚1.232062656552561 1loss/SRqqt3/detection_loss‚1.0862444825656712 /loss/FRt2/assignment_loss‚3.766853988170624 1loss/SRqqt4/assignment_loss‚1.419683252993439 0loss/SRqqt2/detection_loss‚2.335174060636951 epoch‚6 1loss/FBt1/assignment_loss‚0.12050787154074903 0loss/FBt2/assignment_loss‚0.2022906296760567 /loss/FRt3/detection_loss‚1.9185970960184933 0loss/SRqqt4/detection_loss‚2.519571304321289 .loss/FBt2/detection_loss‚2.242289464920759 1loss/FBt3/assignment_loss‚0.13575044660829008 0loss/SRqqt1/detection_loss‚1.951503809156089 _runtime‚9136.937150995 _step‚637 0loss/FRt3/assignment_loss‚1.5790430545806884 2loss/SRqqt3/assignment_loss‚0.9136479856050573 0loss/FBt4/assignment_loss‚0.3892026714590732 /loss/FBt1/detection_loss‚0.6570605832710177ýÜV›×’±"‚(:‡ „ØéÔ“ÖÉn gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚144.489' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.12225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â_m²"‚(:â „ØéԼ̋o disk.rbd0.in‚0 disk.sdb1.in‚0) proc.memory.availableMB‚ 1315.58203125 disk.sda2.in‚8.515625 disk.sdb2.in‚3.78125 disk.sda1.in‚0) proc.memory.percent‚18.33789348602295 disk.dm-0.out‚ 137882.6875# disk.nvme0n1.out‚99310.62890625 disk.nvme1n1.in‚ 744.32421875 disk.md0p1.in‚ 12.296875 disk.md0p1.out‚ 981.1328125) disk./.usagePercent‚75.80446690468746% disk./.usageGB‚2711.1349143981934! disk.sdb.out‚1019.92431640625" disk.sda2.out‚1019.92431640625 disk.md0.out‚ 981.1328125 disk.sda.in‚8.515625! disk.sda.out‚1019.92431640625$ memory_percent‚91.97032451629639 network.recv‚ 2.979064e+06 disk.rbd2.out‚ 2.69140625" disk.sdb2.out‚1019.92431640625# proc.memory.rssMB‚ 3004.48046875 cpu‚25.19047582281794# disk.nvme1n1.out‚38572.05859375 disk.md0.in‚ 12.296875 proc.cpu.threads‚17 disk.sdb.in‚3.78125 disk.nvme0n1.in‚ 939.5390625 disk.rbd2.in‚0 disk.sdb1.out‚0 network.sent‚ 2.0144047e+07 disk.dm-0.in‚ 1683.86328125 disk.rbd0.out‚ 2.466796875 disk.sda1.out‚0•Šj³"‚(:ß “ØéÔðäËl disk.sdb2.in‚3.78125# disk.nvme1n1.out‚38658.87109375 disk.rbd2.out‚ 2.69140625 disk.sdb1.out‚0" disk.nvme0n1.out‚ 99520.5390625 network.sent‚ 2.02516e+07) disk./.usagePercent‚75.80443842632539 disk.sda1.in‚0 disk.md0.out‚ 983.16796875( proc.memory.availableMB‚ 1315.4296875) proc.memory.percent‚18.33789348602295 disk.rbd0.out‚ 2.466796875 disk.sdb.in‚3.78125 disk.sda2.in‚8.515625$ memory_percent‚91.97125434875488 network.recv‚ 2.9843e+06 disk.rbd0.in‚0 disk.nvme1n1.in‚ 744.32421875# proc.memory.rssMB‚ 3004.48046875 cpu‚25.190233696863096 disk.sda.out‚1022.0283203125 disk.sdb1.in‚0 disk.dm-0.in‚ 1683.86328125! disk.sda2.out‚1022.0283203125 disk.nvme0n1.in‚ 939.5390625 disk.sdb.out‚1022.0283203125! disk.sdb2.out‚1022.0283203125 disk.sda1.out‚0 disk.md0p1.in‚ 12.296875 disk.md0p1.out‚ 983.16796875! disk.dm-0.out‚138179.41015625 disk.md0.in‚ 12.296875 proc.cpu.threads‚17% disk./.usageGB‚2711.1338958740234 disk.rbd2.in‚0 disk.sda.in‚8.515625Üè•‘´"‚(:† “ØéÔœÏàl gpu.0.gpu‚23.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚182.364' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.591 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²ºsÒ“µ"‚(:ˆ ¢ØéÔÏÍm gpu.0.gpu‚17.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚182.031' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.50775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õØ{b¶"‚(:× ¢ØéÔÄàëm disk.rbd0.out‚ 2.466796875 disk.sdb.in‚3.78125 disk.nvme1n1.in‚ 744.32421875 disk.sdb2.out‚ 1023.51171875 disk.sda2.in‚8.515625 disk.nvme0n1.in‚ 939.5390625" disk.nvme1n1.out‚ 38745.6640625 disk.md0.in‚ 12.296875 disk.md0.out‚ 984.578125) proc.memory.percent‚18.33789348602295% disk./.usageGB‚2711.1338958740234 disk.sdb2.in‚3.78125 disk.sda.out‚ 1023.51171875 disk.sdb1.in‚0# disk.nvme0n1.out‚99677.91796875 proc.cpu.threads‚17 network.recv‚ 2.989293e+06! disk.dm-0.out‚138423.58203125) proc.memory.availableMB‚ 1315.76171875 network.sent‚ 2.0257881e+07 disk.sda.in‚8.515625 disk.sda1.out‚0 disk.dm-0.in‚ 1683.86328125# proc.memory.rssMB‚ 3004.48046875 cpu‚25.18997086006765 disk.sdb1.out‚0 disk.rbd0.in‚0 disk.md0p1.out‚ 984.578125 disk.sdb.out‚ 1023.51171875 disk.rbd2.out‚ 2.69140625$ memory_percent‚91.96922779083252) disk./.usagePercent‚75.80443842632539 disk.sda1.in‚0 disk.md0p1.in‚ 12.296875 disk.sda2.out‚ 1023.51171875 disk.rbd2.in‚0Ë^Â'2·"- _wandbruntime‚9193  _runtime‚9193ž£hȸ" _runtime‚ 9193.65931045 _step‚638 ' lr-AdamW/pg1‚0.00035473398205174644 ' lr-AdamW/pg2‚0.00035473398205174644 trainer/global_step‚7799 % _timestamp‚1.7885051358820531e+09f§š|͹"Ç _runtime‚ 9193.65931045 _step‚638 ' lr-AdamW/pg1‚0.00035473398205174644 ' lr-AdamW/pg2‚0.00035473398205174644 trainer/global_step‚7799 % _timestamp‚1.7885051358820531e+09þIªŽY2º"- _wandbruntime‚9193  _runtime‚9193!Qcü'»"öO RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 145029936 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 145029287 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8902039019066 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚657163349078528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47301450 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97728484 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚666097868056576 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚734381334033920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 360672581 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259013632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 360646235 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚725050644405760 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9330713568256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192óoE°¼"ûO ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚657163349078528 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚725050620465664 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 197537515 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚666098796030976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47300803 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚635420139818576 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚657163349078528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47301450 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97728484 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚666097868056576 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚734381334033920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 LDDeviceStatsMonitor.on_trainî<¦Q_batch_start/segment.small_pool.allocated‚570 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 360672581 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259013632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 360646235 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚725050644405760 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9330713568256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 145029936 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 145029287 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8902039019066 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 _step‚640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 360717598 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚666181536590336 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 360691226 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97740551 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97740549 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚658 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚725140846170112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8936568425984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 % _timestamp‚1.7885051370349026e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8935658441216 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 360717598 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚734472983906304 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 163128882 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚657244968164352 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 360691226 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163128880 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 163128880 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145047227 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚666180608602624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9331880833024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚644401405730460 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8903156292552 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚658 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚725140846170112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8936568425984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 % _timestamp‚1.7885051370349026e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8935658441216 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 360717598 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚734472983906304 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 163128882 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚657244968164352 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 360691226 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163128880 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 163128880 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145047227 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚666180608602624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9331880833024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚644401405730460 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8903156292552 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 _step‚640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 360717598 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚666181536590336 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 360691226 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97740551 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97740549 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 P^Â"Ø 2loss/SRqqt4/assignment_loss‚1.9358663322547307 1loss/SRqqt3/detection_loss‚0.8605665767654067 _runtime‚9194.813929416 1loss/FBt1/assignment_loss‚0.14963030708635314 0loss/FBt3/assignment_loss‚0.1748912545485187 .loss/FRt1/detection_loss‚2.369709573853195 /loss/FRt3/detection_loss‚1.5615546355644863 .loss/FRt4/detection_loss‚3.802351800600688 .loss/FBt2/detection_loss‚2.680783846161582 %loss/total_loss‚44.17509543312693 .loss/FRt2/detection_loss‚2.363042697682977 1loss/SRqqt1/detection_loss‚2.4320062135948852 epoch‚6 trainer/global_step‚7799 /loss/FBt1/detection_loss‚0.8920418916063176 .loss/FBt3/detection_loss‚2.162844283240182 /loss/FBt4/detection_loss‚2.1504728264278836 2loss/SRqqt1/assignment_loss‚1.2743444569847162 1loss/SRqqt2/assignment_loss‚2.058146482259035 1loss/FBt2/assignment_loss‚0.33365669429853867 1loss/SRqqt2/detection_loss‚2.5517513751983643 % _timestamp‚1.7885051370368912e+09 0loss/FRt3/assignment_loss‚1.1989218946816265 0loss/FBt4/assignment_loss‚0.1705174987065995 _step‚641 0loss/FRt1/assignment_loss‚2.2374391514201495 /loss/FRt2/assignment_loss‚3.598436200618744 0loss/FRt4/assignment_loss‚4.0380927165349325 2loss/SRqqt3/assignment_loss‚0.9269322350398094 0loss/SRqqt4/detection_loss‚2.251094487997202[.¬“Ã"‚(:ˆ ±ØéÔ½ï¼l gpu.0.gpu‚25.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚178.449' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.61225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÔiY0dÄ"‚(:Ù ±ØéÔ«€úm disk.rbd0.in‚0 disk.md0p1.out‚ 985.97265625 disk.sdb.out‚ 1024.96484375# disk.nvme0n1.out‚99678.09765625 disk.sda1.in‚0 disk.sdb2.in‚3.78125) proc.memory.availableMB‚ 1315.40234375! disk.nvme1n1.out‚ 38803.796875 disk.md0p1.in‚ 12.296875) disk./.usagePercent‚75.80448343706995 disk.rbd2.out‚ 2.69140625 disk.rbd0.out‚ 2.466796875 disk.sdb1.in‚0 proc.cpu.threads‚17 disk.sdb.in‚3.78125 disk.rbd2.in‚0 disk.sdb2.out‚ 1024.96484375 disk.sda.in‚8.515625 disk.sda1.out‚0! disk.dm-0.out‚138481.89453125$ memory_percent‚91.97142124176025# proc.memory.rssMB‚ 3004.48046875 disk.nvme0n1.in‚ 939.5390625 disk.md0.in‚ 12.296875 disk.sda.out‚ 1024.96484375 disk.sdb1.out‚0) proc.memory.percent‚18.33789348602295 disk.md0.out‚ 985.97265625 network.recv‚ 2.992838e+06% disk./.usageGB‚2711.1355056762695 disk.sda2.in‚8.515625 disk.sda2.out‚ 1024.96484375 disk.nvme1n1.in‚ 744.32421875 disk.dm-0.in‚ 1683.86328125 network.sent‚ 2.0276179e+07 cpu‚25.1896883086402æ*‚jÅ"‚(:ß ÀØéÔó½Çl disk.md0.out‚ 987.95703125 disk.sdb2.in‚3.78125$ memory_percent‚91.97273254394531) disk./.usagePercent‚75.80448109053823 disk.rbd0.in‚0 disk.sda2.in‚8.515625 disk.nvme1n1.in‚ 744.32421875 network.recv‚ 2.998938e+06 disk.sda1.in‚0 disk.md0p1.in‚ 12.296875 disk.md0p1.out‚ 987.95703125 disk.md0.in‚ 12.296875! disk.sdb2.out‚1027.0048828125# proc.memory.rssMB‚ 3004.48046875 disk.sdb.out‚1027.0048828125 disk.nvme0n1.in‚ 939.5390625% proc.memory.availableMB‚ 1315.1875 cpu‚25.189442299277413% disk./.usageGB‚2711.1354217529297 disk.sdb1.in‚0 disk.sda.in‚8.515625 network.sent‚ 2.037325e+07 disk.sdb1.out‚0 disk.dm-0.in‚ 1683.86328125# disk.nvme0n1.out‚99910.09765625 disk.rbd2.in‚0 disk.rbd2.out‚ 2.69140625 disk.sda.out‚1027.0048828125) proc.memory.percent‚18.33789348602295 disk.sda1.out‚0! disk.sda2.out‚1027.0048828125 proc.cpu.threads‚17 disk.rbd0.out‚ 2.466796875 disk.dm-0.out‚138829.5703125 disk.sdb.in‚3.78125# disk.nvme1n1.out‚38919.47265625~cZ’Æ"‚(:‡ ÀØéÔ˜³úl gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚174.562' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.6405 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|;ÕΓÇ"‚(:ˆ ÏØéÔ䜫m gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚127.779' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.94475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jj¢jÈ"‚(:ß ÏØéÔ°›Þm disk.rbd0.out‚ 2.466796875 disk.md0p1.out‚ 989.34375! disk.sdb.out‚1028.46337890625# memory_percent‚91.9501781463623( proc.memory.availableMB‚ 1318.8828125! proc.memory.rssMB‚ 3004.578125 cpu‚25.189182427415354) disk./.usagePercent‚75.78973729852252 disk.md0p1.in‚ 12.296875 disk.nvme0n1.in‚ 939.5390625 network.sent‚ 2.0379055e+07 disk.sda2.in‚8.515625$ disk.nvme0n1.out‚100045.19921875# disk.nvme1n1.out‚38920.01171875 disk.dm-0.out‚138965.2109375 disk.sdb.in‚3.78125 disk.rbd2.out‚ 2.69140625! disk.sda.out‚1028.46337890625 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 1683.86328125" disk.sda2.out‚1028.46337890625 disk.md0.out‚ 989.34375 disk.rbd2.in‚0 disk.sda1.out‚0 network.recv‚ 3.002936e+06 disk.nvme1n1.in‚ 744.32421875 disk.md0.in‚ 12.296875 disk.sda.in‚8.515625 disk.sda1.in‚0$ disk./.usageGB‚2710.608112335205" disk.sdb2.out‚1028.46337890625 disk.sdb1.in‚0 disk.sdb1.out‚0* proc.memory.percent‚18.338489532470703 disk.sdb2.in‚3.78125þ»Q¼“É"‚(:ˆ ÞØéԾײk gpu.0.gpu‚14.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚181.699' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.42475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥.}DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚670234909772800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚670235837747200 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚648328027844946 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚661244303369728 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 164116722 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 145921274 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 _step‚643 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _runtime‚9251.447159912 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98331734 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9389157847552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 362925483 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47590153 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚729561905688064 trainer/global_step‚7849 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47589540 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚639371036585136 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 362899137 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 198808761 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚670234909772800 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 198782417 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8957916274174 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚661244303369728 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 % _timestamp‚1.7885051936694138e+09 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 11953766408b?)(Ð"ûO P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 _step‚643 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _runtime‚9251.447159912 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98331734 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9389157847552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 362925483 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47590153 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚729561905688064 trainer/global_step‚7849 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47589540 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚639371036585136 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 362899137 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 198808761 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚670234909772800 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 198782417 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8957916274174 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚661244303369728 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 % _timestamp‚1.7885051936694138e+09 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 164116720 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 198782417 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚738951314160640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8991534377472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚738951063535616 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚670234909772800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚670235837747200 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚648328027844946 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚661244303369728 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 164116722 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 145921274 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ƒ§/rå2Ñ"- _wandbruntime‚9252  _runtime‚9252’‰5I'Ò"€N ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚639450037140320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 362944128 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _runtime‚9252.490628875 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚739042716123648 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 198833618 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 145939834 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8992655851008 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚729652131392512 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚670318578306560 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8958125572448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9390338313728 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚661325904452608 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 164136882 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚7849 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 164136880 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98343799 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8991745866240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚661325922455552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 362944128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 164136880 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7885051947126446e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚648408162712768 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98343801 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚661325904452608 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9390560791040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚639450054179680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚661325922455552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8991745866240 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚670318578306560 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 362970500 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚649 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚670317650318848 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8992655851008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47595386 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8959033547660 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 198807248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47596033 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚729652155332608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 362970500 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 164136882 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 _step‚644 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145939185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0ƒøg̽Ó"…N % _timestamp‚1.7885051947126446e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚648408162712768 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98343801 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚661325904452608 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9390560791040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚639450054179680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚661325922455552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8991745866240 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚670318578306560 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 362970500 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.IDÖÉN peak‚ 13684416521 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚649 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚670317650318848 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8992655851008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47595386 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8959033547660 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 198807248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47596033 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚729652155332608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 362970500 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 164136882 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 _step‚644 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145939185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚639450037140320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 362944128 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _runtime‚9252.490628875 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚739042716123648 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 198833618 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 145939834 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8992655851008 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚729652131392512 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚670318578306560 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8958125572448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9390338313728 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚661325904452608 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 164136882 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚7849 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 164136880 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98343799 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8991745866240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚661325922455552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 362944128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 164136880 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0„;¶Ù:2Ô"- _wandbruntime‚9252  _runtime‚9252iÓ‰ZÕ"Ô 0loss/FRt2/assignment_loss‚3.2258474410966387 /loss/FRt3/detection_loss‚1.7556872357644382 0loss/SRqqt4/detection_loss‚2.338478438813111 _runtime‚9252.492366913 1loss/FBt1/assignment_loss‚0.11274349408056748 1loss/SRqqt3/detection_loss‚0.8824201835940282 /loss/FBt1/detection_loss‚0.7385240934386347 epoch‚6 _step‚645 /loss/FRt3/assignment_loss‚1.464886611507785 /loss/FRt4/detection_loss‚3.0609273746095855 /loss/FBt3/detection_loss‚2.2294701814651487 0loss/FBt4/assignment_loss‚0.4548744419227081 %loss/total_loss‚41.69147983762116 % _timestamp‚1.7885051947153728e+09 /loss/FRt4/assignment_loss‚3.339718495977336 0loss/FBt3/assignment_loss‚0.0943029586225748 /loss/FRt1/detection_loss‚1.6873122472497117 /loss/FBt2/detection_loss‚2.2855162024497986 -loss/FBt4/detection_loss‚2.16252164542675 0loss/FRt1/assignment_loss‚2.5012430164549087 2loss/SRqqt3/assignment_loss‚1.0620730578381983 2loss/SRqqt4/assignment_loss‚1.9266119607049843 1loss/FBt2/assignment_loss‚0.28863960932994814 /loss/FRt2/detection_loss‚2.3663434883536296 1loss/SRqqt1/assignment_loss‚1.367498260832602 2loss/SRqqt2/assignment_loss‚2.0294309163895936 1loss/SRqqt1/detection_loss‚1.9742956699863556 0loss/SRqqt2/detection_loss‚2.342112811712118 trainer/global_step‚7849\" Ÿ_Ö"Ù 1loss/SRqqt1/assignment_loss‚1.367498260832602 2loss/SRqqt2/assignment_loss‚2.0294309163895936 1loss/SRqqt1/detection_loss‚1.9742956699863556 0loss/SRqqt2/detection_loss‚2.342112811712118 trainer/global_step‚7849 0loss/FRt2/assignment_loss‚3.2258474410966387 /loss/FRt3/detection_loss‚1.7556872357644382 0loss/SRqqt4/detection_loss‚2.338478438813111 _runtime‚9252.492366913 1loss/FBt1/assignment_loss‚0.11274349408056748 1loss/SRqqt3/detection_loss‚0.8824201835940282 /loss/FBt1/detection_loss‚0.7385240934386347 epoch‚6 _step‚645 /loss/FRt3/assignment_loss‚1.464886611507785 /loss/FRt4/detection_loss‚3.0609273746095855 /loss/FBt3/detection_loss‚2.2294701814651487 0loss/FBt4/assignment_loss‚0.4548744419227081 %loss/total_loss‚41.69147983762116 % _timestamp‚1.7885051947153728e+09 /loss/FRt4/assignment_loss‚3.339718495977336 0loss/FBt3/assignment_loss‚0.0943029586225748 /loss/FRt1/detection_loss‚1.6873122472497117 /loss/FBt2/detection_loss‚2.2855162024497986 -loss/FBt4/detection_loss‚2.16252164542675 0loss/FRt1/assignment_loss‚2.5012430164549087 2loss/SRqqt3/assignment_loss‚1.0620730578381983 2loss/SRqqt4/assignment_loss‚1.9266119607049843 1loss/FBt2/assignment_loss‚0.28863960932994814 /loss/FRt2/detection_loss‚2.3663434883536296…a;_‚‘×"‚(:† íØéÔîÄŒn gpu.0.gpu‚23.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚123.712' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.928 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'Ú|iØ"‚(:Þ íØéÔˆôÆn disk.md0.out‚ 992.45703125 disk.sdb1.in‚0( proc.memory.availableMB‚ 1318.8984375! proc.memory.rssMB‚ 3004.578125 cpu‚25.188674014311154# disk.nvme1n1.out‚38969.11328125 disk.sda.in‚8.515625 disk.rbd0.out‚ 2.466796875 disk.sdb1.out‚0 disk.md0p1.in‚ 12.296875 network.recv‚ 3.010613e+06% disk./.usageGB‚2710.1453819274902 disk.sdb.in‚3.78125 disk.nvme0n1.in‚ 939.5390625 disk.sdb2.out‚1031.708984375 disk.sda2.out‚1031.708984375# disk.nvme0n1.out‚100412.8203125 disk.nvme1n1.in‚ 744.32421875 disk.rbd2.in‚0 disk.sda1.out‚0$ memory_percent‚91.95008277893066 proc.cpu.threads‚17( disk./.usagePercent‚75.7767991626551 disk.sdb.out‚1031.708984375 disk.md0.in‚ 12.296875 disk.sdb2.in‚3.78125 disk.sda.out‚1031.708984375 disk.md0p1.out‚ 992.45703125 disk.dm-0.in‚ 1683.86328125 disk.sda1.in‚0! disk.dm-0.out‚139381.93359375* proc.memory.percent‚18.338489532470703 network.sent‚ 2.0388388e+07 disk.rbd2.out‚ 2.69140625 disk.rbd0.in‚0 disk.sda2.in‚8.515625Ä&@ÉrÙ"‚(:ç üØéÔý“Úm disk.sdb.in‚3.78125( disk./.usagePercent‚75.7768543061502 disk.rbd2.out‚ 2.69140625* proc.memory.percent‚18.338489532470703# disk.nvme1n1.out‚38969.33203125" disk.sdb2.out‚1033.50537109375 disk.sda.in‚8.515625 disk.sda1.out‚0 disk.sdb1.out‚0$ memory_percent‚91.94881916046143 disk.sda2.in‚8.515625 disk.md0.out‚ 994.20703125 disk.sdb2.in‚3.78125 disk.md0p1.out‚ 994.20703125 cpu‚25.188401295067184% disk./.usageGB‚2710.1473541259766 disk.md0.in‚ 12.296875 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.md0p1.in‚ 12.296875" disk.sda2.out‚1033.50537109375# disk.nvme0n1.out‚100741.8828125 disk.nvme1n1.in‚ 744.32421875 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1683.86328125! proc.memory.rssMB‚ 3004.578125 network.sent‚ 2.0498383e+07 network.recv‚ 3.016359e+06! disk.sda.out‚1033.50537109375 proc.cpu.threads‚17 disk.rbd0.out‚ 2.466796875! disk.sdb.out‚1033.50537109375) proc.memory.availableMB‚ 1319.10546875 disk.nvme0n1.in‚ 939.5390625! disk.dm-0.out‚139711.21484375j2“Ú"‚(:ˆ üØéÔ¶¾‰o gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#ªQ;Û"‚(:… ‹ÙéÔÇŽ’m gpu.0.gpu‚7.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚128.056' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 !…ƒdÜ"‚(:Ù ‹ÙéÔß›ûm) disk./.usagePercent‚75.77860332555592 disk.rbd0.in‚0 disk.rbd0.out‚ 2.466796875 disk.sda2.in‚8.515625$ disk.nvme0n1.out‚100835.78515625 disk.sdb2.in‚3.78125 disk.sda.out‚ 1035.33984375 disk.sdb1.in‚0 disk.md0p1.out‚ 995.97265625 disk.sdb2.out‚ 1035.33984375$ memory_percent‚91.94884300231934 cpu‚25.18814996466331 proc.cpu.threads‚17 disk.dm-0.in‚ 1683.86328125! disk.dm-0.out‚139805.39453125 disk.sdb.in‚3.78125 disk.nvme1n1.in‚ 744.32421875 disk.rbd2.in‚0 network.recv‚ 3.020556e+06 disk.sda.in‚8.515625 disk.sdb1.out‚0! disk.nvme1n1.out‚ 38969.609375 disk.md0.out‚ 995.97265625! proc.memory.rssMB‚ 3004.578125 disk.sda1.in‚0 disk.md0.in‚ 12.296875( proc.memory.availableMB‚ 1319.1015625 disk.sdb.out‚ 1035.33984375 disk.nvme0n1.in‚ 939.5390625 disk.rbd2.out‚ 2.69140625% disk./.usageGB‚2710.2099075317383 disk.sda1.out‚0 disk.md0p1.in‚ 12.296875 disk.sda2.out‚ 1035.33984375* proc.memory.percent‚18.338489532470703 network.sent‚ 2.0504202e+07%kd“Ý"‚(:ˆ šÙéÔ‰÷»m gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.461' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.11525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>ׂDhÞ"‚(:Ý šÙéÔ»†¢n network.recv‚ 3.025319e+06 disk.nvme1n1.in‚ 744.32421875! disk.sda.out‚1036.86376953125 disk.sda1.in‚0 disk.sdb1.out‚0 disk.md0p1.out‚ 997.4296875 disk.dm-0.out‚140020.5078125" disk.nvme1n1.out‚ 38969.7578125 disk.md0.out‚ 997.4296875 disk.md0p1.in‚ 12.296875 disk.sda2.in‚8.515625 disk.md0.in‚ 12.296875 disk.rbd0.in‚0# memory_percent‚91.9487714767456* proc.memory.percent‚18.338489532470703" disk.sda2.out‚1036.86376953125 disk.sdb2.in‚3.78125 disk.sda1.out‚0) proc.memory.availableMB‚ 1319.11328125! proc.memory.rssMB‚ 3004.578125$ disk./.usageGB‚2710.397804260254 disk.sdb1.in‚0 disk.sdb.in‚3.78125 disk.nvme0n1.out‚ 101050.75" disk.sdb2.out‚1036.86376953125 disk.dm-0.in‚ 1683.86328125 proc.cpu.threads‚17 network.sent‚ 2.0510576e+07! disk.sdb.out‚1036.86376953125 disk.rbd2.in‚0 disk.sda.in‚8.515625) disk./.usagePercent‚75.78385699672612 disk.nvme0n1.in‚ 939.5390625 disk.rbd2.out‚ 2.69140625 disk.rbd0.out‚ 2.466796875 cpu‚25.1878992207854866“¹’2ß"- _wandbruntime‚9309  _runtime‚9309†vŸ(Éà"à ' lr-AdamW/pg1‚0.00034044454031057907 ' lr-AdamW/pg2‚0.00034044454031057907 trainer/global_step‚7899 % _timestamp‚1.7885052513270454e+09 _runtime‚9309.104399573 _step‚646V+Îá"È % _timestamp‚1.7885052513270454e+09 _runtime‚9309.104399573 _step‚646 ' lr-AdamW/pg1‚0.00034044454031057907 ' lr-AdamW/pg2‚0.00034044454031057907 trainer/global_step‚7899†rDž2â"- _wandbruntime‚9309  _runtime‚9309–ïµ|ü'ã"öO JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚622 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47878945 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚643321899272976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 165124720 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚665325257660928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚674372879463424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98934984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47878325 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 365152039 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 165124722 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚620 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9012885554278 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 146813931 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9046711831040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚643321916312336 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚647 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚674371951489024 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚9309.105791589 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 200027319 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚665325239657984 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9448074988544 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚652334784827254 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 200027319 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 % _timestamp‚1.7885052513283768e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165124720 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 200053663 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9013793529282 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚665325257660928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98934986 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0â†ldä"ûO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 200053663 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚622 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47878945 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚643321899272976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 165124720 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚665325257660928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚674372879463424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98934984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47878325 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 365152039 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 165124722 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚620 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9012885554278 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 146813931 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9046711831040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚643321916312336 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚647 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚674371951489024 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚9309.105791589 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 200027319 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚665325239657984 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9448074988544 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚652334784827254 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 200027319 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 % _timestamp‚1.7885052513283768e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165124720 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 200053663 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9013793529282 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚665325257660928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98934986 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 365197030 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚674455620022784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9048743276032 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 165144880 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 200078520 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146830982 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98947051 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚743612693023744 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9449252468736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9449025797120 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚665406876746752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚665406858743808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 200052150 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚652414919695076 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 165144880 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚625 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 % _timestamp‚1.7885052524036748e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 365223402 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚665406876746752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 365197030 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98947049 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚674455620022784 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚734163416614912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47884558 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9014002827556 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _step‚648 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165144882 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9047833291264 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 165144882 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 146831609 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 200052150 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚652414919695076 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 165144880 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚625 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 % _timestamp‚1.7885052524036748e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 365223402 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚665406876746752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 365197030 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98947049 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚674455620022784 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚734163416614912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47884558 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9014002827556 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _step‚648 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165144882 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9047833291264 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 165144882 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 146831609 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 365197030 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚674455620022784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9048743276032 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 165144880 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 200078520 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146830982 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98947051 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚743612693023744 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9449252468736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9449025797120 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚665406876746752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚665406858743808ˆ„×Ãý2è"- _wandbruntime‚9310  _runtime‚9310GQÏÐ\é"Ö /loss/FBt3/detection_loss‚1.9381228685379028 2loss/SRqqt2/assignment_loss‚1.5626710399985313 epoch‚6 % _timestamp‚1.7885052524079654e+09 2loss/SRqqt1/assignment_loss‚1.2763282724011402 2loss/SRqqt4/assignment_loss‚1.5797920051429952 0loss/FBt2/assignment_loss‚0.2078432086560598 0loss/FBt4/assignment_loss‚0.7882532077440244 1loss/SRqqt1/detection_loss‚1.8642403277067037 /loss/FRt2/assignment_loss‚3.549872029086818 .loss/FRt4/detection_loss‚2.555340511724353 1loss/SRqqt2/detection_loss‚2.8458655402064323 _step‚649 /loss/FRt2/detection_loss‚2.2482415556745683 /loss/FBt4/detection_loss‚3.6246097130434856 %loss/total_loss‚41.02970709146918 0loss/FRt3/assignment_loss‚1.1292346942424774 1loss/SRqqt3/detection_loss‚0.9700185266789049 trainer/global_step‚7899 0loss/FRt1/assignment_loss‚1.7993629319327218 1loss/FBt1/assignment_loss‚0.11534457360139641 2loss/FBt3/assignment_loss‚0.061366510577499866 .loss/FRt3/detection_loss‚2.003800257220864 /loss/FRt4/assignment_loss‚2.982816580682993 /loss/FRt1/detection_loss‚1.7417576822787981 0loss/SRqqt4/detection_loss‚2.546495388661112 _runtime‚9310.185127522 2loss/SRqqt3/assignment_loss‚0.8437587282814396 /loss/FBt1/detection_loss‚0.7209409123985097 .loss/FBt2/detection_loss‚2.073630024989446qš¤'aê"Û %loss/total_loss‚41.02970709146918 0loss/FRt3/assignment_loss‚1.1292346942424774 1loss/SRqqt3/detection_loss‚0.9700185266789049 trainer/global_step‚7899 0loss/FRt1/assignment_loss‚1.7993629319327218 1loss/FBt1/assignment_loss‚0.11534457360139641 2loss/FBt3/assignment_loss‚0.061366510577499866 .loss/FRt3/detection_loss‚2.003800257220864 /loss/FRt4/assignment_loss‚2.982816580682993 /loss/FRt1/detection_loss‚1.7417576822787981 0loss/SRqqt4/detection_loss‚2.546495388661112 _runtime‚9310.185127522 2loss/SRqqt3/assignment_loss‚0.8437587282814396 /loss/FBt1/detection_loss‚0.7209409123985097 .loss/FBt2/detection_loss‚2.073630024989446 /loss/FBt3/detection_loss‚1.9381228685379028 2loss/SRqqt2/assignment_loss‚1.5626710399985313 epoch‚6 % _timestamp‚1.7885052524079654e+09 2loss/SRqqt1/assignment_loss‚1.2763282724011402 2loss/SRqqt4/assignment_loss‚1.5797920051429952 0loss/FBt2/assignment_loss‚0.2078432086560598 0loss/FBt4/assignment_loss‚0.7882532077440244 1loss/SRqqt1/detection_loss‚1.8642403277067037 /loss/FRt2/assignment_loss‚3.549872029086818 .loss/FRt4/detection_loss‚2.555340511724353 1loss/SRqqt2/detection_loss‚2.8458655402064323 _step‚649 /loss/FRt2/detection_loss‚2.2482415556745683 /loss/FBt4/detection_loss‚3.6246097130434856‰{;\µ’ë"‚(:‡ ©ÙéÔÛ¸m gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.942' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4855 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M§î|qì"‚(:æ ©ÙéÔ«¼‰o# disk.nvme1n1.out‚38970.05078125 disk.rbd2.out‚ 2.69140625" disk.sdb2.out‚1038.85986328125 disk.rbd0.in‚0! proc.memory.rssMB‚ 3004.578125 disk.sda1.in‚0 disk.md0p1.in‚ 12.296875) proc.memory.availableMB‚ 1319.12109375# disk.nvme0n1.out‚101404.9609375 disk.md0p1.out‚ 999.35546875* proc.memory.percent‚18.338489532470703 network.sent‚ 2.0526515e+07 disk.sda.in‚8.515625 disk.sdb1.in‚0 disk.rbd0.out‚ 2.466796875 network.recv‚ 3.028825e+06 disk.dm-0.in‚ 1683.86328125 disk.sda2.in‚8.515625" disk.sda2.out‚1038.85986328125 disk.md0.in‚ 12.296875 disk.rbd2.in‚0 cpu‚25.18761966789867! disk.sdb.out‚1038.85986328125 disk.nvme1n1.in‚ 744.32421875 disk.md0.out‚ 999.35546875 disk.sdb2.in‚3.78125$ memory_percent‚91.94872379302979) disk./.usagePercent‚75.78384345083855! disk.dm-0.out‚140375.01171875! disk.sda.out‚1038.85986328125 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sdb.in‚3.78125 disk.nvme0n1.in‚ 939.5390625 disk.sdb1.out‚0$ disk./.usageGB‚2710.397319793701¤¯sô“í"‚(:ˆ ¸ÙéÔ؃‹m gpu.0.gpu‚38.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚133.349' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.33725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qV^‚tî"‚(:é ¸ÙéÔˆàõm disk.sdb1.out‚0 network.sent‚ 2.0622619e+07# disk.nvme1n1.out‚38970.24609375 disk.sdb1.in‚0 disk.md0p1.out‚ 1000.66796875 disk.nvme0n1.in‚ 939.5390625 disk.nvme1n1.in‚ 744.32421875 disk.rbd2.in‚0" disk.sda2.out‚1040.23095703125* proc.memory.percent‚18.338489532470703! disk.dm-0.out‚140588.45703125 disk.sdb.in‚3.78125 disk.sda.in‚8.515625! disk.sda.out‚1040.23095703125 disk.sda1.in‚0$ memory_percent‚91.94872379302979 network.recv‚ 3.033629e+06" disk.sdb2.out‚1040.23095703125 disk.sda2.in‚8.515625) proc.memory.availableMB‚ 1319.12109375! disk.sdb.out‚1040.23095703125 proc.cpu.threads‚17# disk.nvme0n1.out‚101618.2109375 disk.md0.in‚ 12.296875 disk.sdb2.in‚3.78125! proc.memory.rssMB‚ 3004.578125 disk.dm-0.in‚ 1683.86328125) disk./.usagePercent‚75.79083142226104$ disk./.usageGB‚2710.647243499756 disk.md0.out‚ 1000.66796875 disk.rbd2.out‚ 2.69140625 disk.md0p1.in‚ 12.296875 cpu‚25.187378807720123 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 2.466796875{™œÏ“ï"‚(:ˆ ÇÙéÔ€ÌÀm gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚117.193' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.29825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘P 9ð"‚(:ã ÇÙéԛ߀n disk.md0p1.in‚ 12.296875! disk.sda.out‚1041.58837890625 disk.sda1.out‚0 cpu‚25.18713172990595 disk.sdb1.in‚0 disk.md0p1.out‚ 1001.97265625 disk.nvme1n1.out‚ 38970.4375 disk.md0.in‚ 12.296875( proc.memory.availableMB‚ 1319.1171875 network.recv‚ 3.037479e+06) disk./.usagePercent‚75.79083142226104" disk.sdb2.out‚1041.58837890625 disk.rbd0.in‚0 disk.dm-0.in‚ 1683.86328125# memory_percent‚91.9487476348877* proc.memory.percent‚18.338489532470703 disk.md0.out‚ 1001.97265625 disk.rbd2.in‚0! proc.memory.rssMB‚ 3004.578125 proc.cpu.threads‚17 disk.nvme0n1.in‚ 939.5390625 disk.sda.in‚8.515625! disk.dm-0.out‚140590.43359375 disk.sdb.in‚3.78125! disk.sdb.out‚1041.58837890625 disk.sda2.in‚8.515+Uzf5625" disk.sda2.out‚1041.58837890625 network.sent‚ 2.0628437e+07 disk.sda1.in‚0$ disk.nvme0n1.out‚101619.99609375 disk.nvme1n1.in‚ 744.32421875 disk.rbd0.out‚ 2.466796875 disk.sdb1.out‚0$ disk./.usageGB‚2710.647243499756 disk.rbd2.out‚ 2.69140625 disk.sdb2.in‚3.78125õ'›¿’ñ"‚(:‡ ÖÙéÔîšûl gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚174.562' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.6405 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­Kòsò"‚(:è ÖÙéÔчn disk.sda2.in‚8.515625 disk.rbd0.out‚ 2.466796875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0p1.out‚ 1003.83984375* proc.memory.percent‚18.338489532470703 cpu‚25.18685898349758 network.recv‚ 3.042311e+06 disk.md0.out‚ 1003.83984375! disk.sda.out‚1043.51904296875 disk.sda1.in‚0" disk.sdb2.out‚1043.51904296875 disk.sda.in‚8.515625 proc.cpu.threads‚17) disk./.usagePercent‚75.79083142226104! disk.sdb.out‚1043.51904296875" disk.nvme1n1.out‚ 38970.6484375! proc.memory.rssMB‚ 3004.578125 network.sent‚ 2.0634765e+07 disk.nvme0n1.in‚ 939.5390625$ disk./.usageGB‚2710.647243499756 disk.rbd2.out‚ 2.69140625 disk.sda1.out‚0 disk.md0p1.in‚ 12.296875! disk.dm-0.out‚141015.80859375" disk.sda2.out‚1043.51904296875$ disk.nvme0n1.out‚102045.16015625 disk.nvme1n1.in‚ 744.32421875 disk.md0.in‚ 12.296875 disk.rbd2.in‚0 disk.sdb2.in‚3.78125 disk.rbd0.in‚0$ memory_percent‚91.94886684417725 disk.sdb.in‚3.78125 disk.dm-0.in‚ 1683.86328125) proc.memory.availableMB‚ 1319.09765625<2„×2ó"- _wandbruntime‚9367  _runtime‚9367¥·wÉô"à ' lr-AdamW/pg1‚0.00033335713641261417 ' lr-AdamW/pg2‚0.00033335713641261417 trainer/global_step‚7949 % _timestamp‚1.7885053094353669e+09 _runtime‚9367.212801023 _step‚650‹s(Îõ"È trainer/global_step‚7949 % _timestamp‚1.7885053094353669e+09 _runtime‚9367.212801023 _step‚650 ' lr-AdamW/pg1‚0.00033335713641261417 ' lr-AdamW/pg2‚0.00033335713641261417Š «/2ö"- _wandbruntime‚9367  _runtime‚9367Öùû'÷"õO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚7949 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9103709227520 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚647272796039536 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9103709227520 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚9367.214672461 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚669406211952128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚678508993205248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48166929 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9069670784390 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 166132722 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 166132720 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚656341541809562 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 367404941 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48167552 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚678509921179648 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 147705788 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚669406193949184 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9102799256064 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 201298565 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚738584476132864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9068762809386 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615ßz^(ø"úO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 147705788 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚669406193949184 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9102799256064 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 201298565 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚738584476132864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9068762809386 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚7949 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9103709227520 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚647272796039536 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9103709227520 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚9367.214672461 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚669406211952128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚678508993205248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48166929 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9069670784390 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 166132722 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 166132720 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚656341541809562 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 367404941 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48167552 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚678509921179648 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1‹õj‚_2ù"- _wandbruntime‚9368  _runtime‚93686rQ&ú"€N UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚678591733751296 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚656422601691956 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9104830701056 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚669487831037952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚647351813634080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9070788057876 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 201297052 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48173294 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚7949 % _timestamp‚1.7885053103938606e+09 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9103920716288 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 166152880 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 367449932 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚738674701837312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚656421676677384 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚669487831037952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99550301 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚748182384189440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9069880082664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚669487813035008 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚748182127286272 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 367449932 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 201297052 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚738674725777408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9103920716288 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 166152880 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99550299 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚678592661739008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48172651 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9507425448960 _runtime‚9368.171796584 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147722950 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚669487813035008 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 367476304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚678592661739008 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚643 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.curœfž¸µrent‚0 _step‚652 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 367476304R$ñ 'û"…N T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚678592661739008 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚643 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚652 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 367476304 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚678591733751296 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚656422601691956 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9104830701056 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚669487831037952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚647351813634080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9070788057876 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 201297052 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48173294 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚7949 % _timestamp‚1.7885053103938606e+09 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9103920716288 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 166152880 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 367449932 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚738674701837312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚656421676677384 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚669487831037952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99550301 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚748182384189440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9069880082664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚669487813035008 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚748182127286272 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 367449932 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 201297052 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚738674725777408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9103920716288 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 166152880 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99550299 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚678592661739008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48172651 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9507425448960 _runtime‚9368.171796584 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147722950 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚669487813035008 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 367476304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856Œió)€2ü"- _wandbruntime‚9368  _runtime‚9368Éh^Wý"Ñ 1loss/SRqqt3/detection_loss‚1.0172602181836705 -loss/FBt2/detection_loss‚2.70916764031757 _runtime‚9368.173334371 /loss/FRt2/assignment_loss‚3.233674059808254 2loss/SRqqt3/assignment_loss‚0.9723613778177392 1loss/FBt4/assignment_loss‚0.43743851229325326 .loss/FRt4/detection_loss‚3.151043243408203 /loss/FRt4/assignment_loss‚2.234125943183899 2loss/SRqqt2/assignment_loss‚1.5640521577359843 0loss/FRt1/assignment_loss‚2.1713764117314267 1loss/FBt3/assignment_loss‚0.18657853491902188 1loss/SRqqt2/detection_loss‚2.2183314796004976 epoch‚6 ,loss/FRt3/detection_loss‚1.9301906188653 0loss/SRqqt4/detection_loss‚2.571735522963784 /loss/FBt3/detection_loss‚3.4285569861531258 2loss/SRqqt1/assignment_loss‚1.2341810883954167 _step‚653 0loss/FRt3/assignment_loss‚1.8122051021327144 0loss/FBt1/assignment_loss‚0.1765147492445889 1loss/FBt2/assignment_loss‚0.13179146138777884 /loss/FBt4/detection_loss‚2.0699970642725627 trainer/global_step‚7949 # _timestamp‚1.78850531039634e+09 /loss/FRt1/detection_loss‚2.0782604009579395 1loss/SRqqt1/detection_loss‚1.7185568500842368 /loss/FBt1/detection_loss‚0.6232095807854545 &loss/total_loss‚41.668938440725114 2loss/SRqqt4/assignment_loss‚1.6272978653961963 .loss/FRt2/detection_loss‚2.371031571086496"Õb\þ"Ö /loss/FRt1/detection_loss‚2.0782604009579395 1loss/SRqqt1/detection_loss‚1.7185568500842368 /loss/FBt1/detection_loss‚0.6232095807854545 &loss/total_loss‚41.668938440725114 2loss/SRqqt4/assignment_loss‚1.6272978653961963 .loss/FRt2/detection_loss‚2.371031571086496 1loss/SRqqt3/detection_loss‚1.0172602181836705 -loss/FBt2/detection_loss‚2.70916764031757 _runtime‚9368.173334371 /loss/FRt2/assignment_loss‚3.233674059808254 2loss/SRqqt3/assignment_loss‚0.9723613778177392 1loss/FBt4/assignment_loss‚0.43743851229325326 .loss/FRt4/detection_loss‚3.151043243408203 /loss/FRt4/assignment_loss‚2.234125943183899 2loss/SRqqt2/assignment_loss‚1.5640521577359843 0loss/FRt1/assignment_loss‚2.1713764117314267 1loss/FBt3/assignment_loss‚0.18657853491902188 1loss/SRqqt2/detection_loss‚2.2183314796004976 epoch‚6 ,loss/FRt3/detection_loss‚1.9301906188653 0loss/SRqqt4/detection_loss‚2.571735522963784 /loss/FBt3/detection_loss‚3.4285569861531258 2loss/SRqqt1/assignment_loss‚1.2341810883954167 _step‚653 0loss/FRt3/assignment_loss‚1.8122051021327144 0loss/FBt1/assignment_loss‚0.1765147492445889 1loss/FBt2/assignment_loss‚0.13179146138777884 /loss/FBt4/detection_loss‚2.0699970642725627 trainer/global_step‚7949 # _timestamp‚1.78850531039634e+09 ÌÝ“ÿ"‚(:ˆ åÙéÔm gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚167.157' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.78925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/ˆ n€#‚(:ã åÙéÔ§ÄÄn network.recv‚ 3.045672e+06 disk.rbd2.in‚0! disk.sda.out‚1045.01513671875 disk.sda1.in‚0* proc.memory.percent‚18.338489532470703 network.sent‚ 2.0650662e+07# disk.nvme1n1.out‚38970.89453125 disk.sda1.out‚0 disk.md0p1.out‚ 1005.28125! disk.sdb.out‚1045.01513671875$ disk./.usageGB‚2710.647300720215 disk.dm-0.in‚ 1683.86328125 disk.dm-0.out‚141188.8046875$ disk.nvme0n1.out‚102217.91015625 disk.sda2.in‚8.515625" disk.sda2.out‚1045.01513671875 disk.rbd2.out‚ 2.69140625 disk.sdb1.in‚0 disk.md0p1.in‚ 12.296875) disk./.usagePercent‚75.79083302216901 disk.md0.in‚ 12.296875 disk.sdb1.out‚0 disk.nvme0n1.in‚ 939.5390625 proc.cpu.threads‚17 disk.sdb2.in‚3.78125 cpu‚25.186637895134943 disk.rbd0.out‚ 2.466796875$ memory_percent‚91.95053577423096) proc.memory.availableMB‚ 1318.82421875! proc.memory.rssMB‚ 3004.578125 disk.nvme1n1.in‚ 744.32421875 disk.md0.out‚ 1005.28125" disk.sdb2.out‚1045.01513671875 disk.sda.in‚8.515625 disk.rbd0.in‚0 disk.sdb.in‚3.78125rÓ´“#‚(:ˆ ôÙéÔ¯“¾m gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚135.517' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.87925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù¤iÖf‚#‚(:Û ôÙéÔ§¶øm disk.rbd2.out‚ 2.69140625! disk.dm-0.out‚141456.39453125 cpu‚25.18637043883786 disk.rbd0.in‚0 disk.sdb1.out‚0) proc.memory.availableMB‚ 1318.80078125 network.recv‚ 3.051412e+06) disk./.usagePercent‚75.77608816354797 disk.sda2.in‚8.515625 disk.md0p1.in‚ 12.296875 disk.sdb.in‚3.78125 disk.nvme1n1.in‚ 744.32421875 disk.sdb2.out‚ 1046.60546875 disk.rbd0.out‚ 2.466796875 disk.md0p1.out‚ 1006.81640625$ memory_percent‚91.95067882537842 network.sent‚ 2.0747546e+07 disk.nvme1n1.out‚ 38971.09375 disk.sdb2.in‚3.78125 disk.sdb.out‚ 1046.60546875$ disk.nvme0n1.out‚102485.30078125 disk.md0.in‚ 12.296875 disk.md0.out‚ 1006.81640625 disk.sda.out‚ 1046.60546875! proc.memory.rssMB‚ 3004.578125* proc.memory.percent‚18.338489532470703 proc.cpu.threads‚17 disk.dm-0.in‚ 1683.86328125 disk.nvme0n1.in‚ 939.5390625 disk.rbd2.in‚0 disk.sda.in‚8.515625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb1.in‚0% disk./.usageGB‚2710.1199531555176 disk.sda2.out‚ 1046.60546875Ðå³ï’ƒ#‚(:‡ ƒÚéÔš””m gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚158.918' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.7295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸µq„#‚(:æ ƒÚéÔ³†ùm network.recv‚ 3.055691e+06( disk./.usagePercent‚75.7760862436584 disk.md0p1.out‚ 1008.13671875 disk.dm-0.in‚ 1683.86328125! disk.sda.out‚1047.98486328125 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 1008.13671875" disk.sdb2.out‚1047.98486328125 disk.sda.in‚8.515625* proc.memory.percent‚18.338489532470703 cpu‚25.18615350895437 network.sent‚ 2.0753886e+07 disk.md0.in‚ 12.296875 disk.rbd2.in‚0 disk.md0p1.in‚ 12.296875! disk.sdb.out‚1047.98486328125# disk.nvme0n1.out‚102485.4609375# disk.nvme1n1.out‚38971.49609375 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 1318.8984375 disk.sda2.in‚8.515625 disk.nvme1n1.in‚ 744.32421875 disk.rbd2.out‚ 2.69140625 disk.rbd0.out‚ 2.466796875 proc.cpu.threads‚17" disk.sda2.out‚1047.98486328125$ disk./.usageGB‚2710.119884490967! disk.dm-0.out‚141456.95703125 disk.nvme0n1.in‚ 939.5390625 disk.sdb2.in‚3.78125$ memory_percent‚91.95008277893066! proc.memory.rssMB‚ 3004.578125 disk.sdb.in‚3.78125¯œ“…#‚(:ˆ ’ÚéÔôÓÇm gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.383' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.09575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Å™/“i†#‚(:Þ ’ÚéÔÞã…n disk.nvme1n1.in‚ 744.32421875! proc.memory.rssMB‚ 3004.578125 disk.sdb1.out‚0 disk.dm-0.in‚ 1683.86328125 disk.dm-0.out‚141901.5703125# disk./.usageGB‚2710.14497756958 disk.sda1.out‚0 disk.md0p1.out‚ 1010.046875 disk.nvme0n1.in‚ 939.5390625 disk.md0.out‚ 1010.046875$ memory_percent‚91.95008277893066! disk.sda.out‚1049.95849609375! disk.sdb.out‚1049.95849609375 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 2.466796875 disk.sdb.in‚3.78125 disk.sda2.in‚8.515625# disk.nvme0n1.out‚102929.9140625( proc.memory.availableMB‚ 1318.8984375* proc.memory.percent‚18.338489532470703) disk./.usagePercent‚75.77678785663868 disk.rbd2.out‚ 2.69140625 disk.sda1.in‚0" disk.sda2.out‚1049.95849609375 proc.cpu.threads‚17" disk.sdb2.out‚1049.95849609375 disk.sda.in‚8.515625 disk.md0p1.in‚ 12.296875 disk.nvme1n1.out‚ 38971.65625 disk.md0.in‚ 12.296875 cpu‚25.18591027863764 network.sent‚ 2.0760281e+07 network.recv‚ 3.060747e+06 disk.sdb2.in‚3.78125 disk.sdb1.in‚0”„¸ç2‡#- _wandbruntime‚9424  _runtime‚9424 áÇȈ# ' lr-AdamW/pg1‚0.00032630991585764216 ' lr-AdamW/pg2‚0.00032630991585764216 trainer/global_step‚7999 $ _timestamp‚1.788505366937761e+09 _runtime‚9424.715173638 _step‚654 _²Í‰#Ç ' lr-AdamW/pg1‚0.00032630991585764216 ' lr-AdamW/pg2‚0.00032630991585764216 trainer/global_step‚7999 $ _timestamp‚1.788505366937761e+09 _runtime‚9424.715173638 _step‚654Žpô™2Š#- _wandbruntime‚9424  _runtime‚9424…Ííýþ'‹#øO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚660349223806234 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚682646962895872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9158886681088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48456909 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚752660800962560 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9159796652544 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 100141486 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 202517123 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 167140720 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9564801565184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 369684189 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48456287 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238042112 _step‚655 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚660348298791870 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚622 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚651223658727376 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 167140722 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚673487166243328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 148598395 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚682646962895872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 148597771 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885053669394188e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9125548039498 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 202517123 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚743095785295360 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚682646034921472 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 trainer/global_step‚7999 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9159796652544 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 167140722 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ¹šæ Œ#ýO trainer/global_step‚7999 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9159796652544 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 167140722 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚660349223806234 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚682646962895872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9158886681088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48456909 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool4feõ.peak‚ 355286016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚752660800962560 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9159796652544 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 100141486 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 202517123 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 167140720 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9564801565184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 369684189 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48456287 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238042112 _step‚655 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚660348298791870 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚622 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚651223658727376 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 167140722 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚673487166243328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 148598395 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚682646962895872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 148597771 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885053669394188e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9125548039498 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 202517123 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚743095785295360 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚682646034921472 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615=ºþ‘2#- _wandbruntime‚9425  _runtime‚9425Òù‚'Ž#‚N N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚673568767326208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚651302693361280 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚682728775467520 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚682729703455232 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100153549 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9160008141312 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 148616134 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 202541954 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 369729206 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100153551 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148615514 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9566196806144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚651302676321920 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 _runtime‚9425.752635903 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 202541954 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚743186010999808 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚660428433659692 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9160918126080 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 369702834 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9565974328832 trainer/global_step‚7999 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 167160882 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚673568785329152 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚673568785329152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9160918126080 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 369729206 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 202568324 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48462583 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚752752207805952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚752751961388544 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚743185987059712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚660429358674264 _step‚656 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 167160882 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885053679747128e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 202568324 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0×ògm '#‡N I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9160918126080 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 369702834 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9565974328832 trainer/global_step‚7999 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 167160882 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚673568785329152 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚673568785329152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9160918126080 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 369729206 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 202568324 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48462583 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚752752207805952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚752751961388544 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚743185987059712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚660429358674264 _step‚656 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 167160882 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885053679747128e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 202568324 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚673568767326208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚651302693361280 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚682728775467520 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚682729703455232 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100153549 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9160008141312 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 148616134 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 202541954 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 369729206 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100153551 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148615514 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9566196806144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚651302676321920 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 _runtime‚9425.752635903 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 202541954 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚743186010999808 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚660428433659692Â’ w2#- _wandbruntime‚9425  _runtime‚9425yÍݯX‘#Ò /loss/FBt4/assignment_loss‚0.151406156877844 /loss/FBt3/detection_loss‚1.8292415589094162 epoch‚6 0loss/FRt4/assignment_loss‚3.6998697356744246 1loss/SRqqt4/assignment_loss‚1.534776202767777 -loss/FBt2/detection_loss‚2.00456170241038 _step‚657 # _timestamp‚1.78850536797723e+09 2loss/SRqqt3/assignment_loss‚1.0267242489072184 /loss/FRt3/detection_loss‚1.7120240136744485 /loss/FRt4/detection_loss‚3.2250489755110308 1loss/SRqqt1/detection_loss‚2.1106827280351093 1loss/SRqqt2/detection_loss‚2.2319770073890686 &loss/total_loss‚41.078878676126166 /loss/FRt1/detection_loss‚2.2385232216935367 /loss/FRt3/assignment_loss‚1.473046606495267 2loss/SRqqt1/assignment_loss‚1.4903583923088652 1loss/FBt1/assignment_loss‚0.12093330741170157 /loss/FBt4/detection_loss‚2.0788589835166933 0loss/FBt3/assignment_loss‚0.0559276859916281 .loss/FRt2/detection_loss‚2.378406275259821 1loss/SRqqt3/detection_loss‚1.0961362290196122 1loss/SRqqt4/detection_loss‚2.4095417141914366 /loss/FBt1/detection_loss‚0.8010401132272645 0loss/FRt1/assignment_loss‚2.5859264704314144 2loss/SRqqt2/assignment_loss‚1.7230221265554428 0loss/FBt2/assignment_loss‚0.3752000416240359 trainer/global_step‚7999 _runtime‚ 9425.75424698 0loss/FRt2/assignment_loss‚2.7256451782427336åŠæF]’#× 2loss/SRqqt1/assignment_loss‚1.4903583923088652 1loss/FBt1/assignment_loss‚0.12093330741170157 /loss/FBt4/detection_loss‚2.0788589835166933 0loss/FBt3/assignment_loss‚0.0559276859916281 .loss/FRt2/detection_loss‚2.378406275259821 1loss/SRqqt3/detection_loss‚1.0961362290196122 1loss/SRqqt4/detection_loss‚2.4095417141914366 /loss/FBt1/detection_loss‚0.8010401132272645 0loss/FRt1/assignment_loss‚2.5859264704314144 2loss/SRqqt2/assignment_loss‚1.7230221265554428 0loss/FBt2/assignment_loss‚0.3752000416240359 trainer/global_step‚7999 _runtime‚ 9425.75424698 0loss/FRt2/assignment_loss‚2.7256451782427336 /loss/FBt4/assignment_loss‚0.151406156877844 /loss/FBt3/detection_loss‚1.8292415589094162 epoch‚6 0loss/FRt4/assignment_loss‚3.6998697356744246 1loss/SRqqt4/assignment_loss‚1.534776202767777 -loss/FBt2/detection_loss‚2.00456170241038 _step‚657 # _timestamp‚1.78850536797723e+09 2loss/SRqqt3/assignment_loss‚1.0267242489072184 /loss/FRt3/detection_loss‚1.7120240136744485 /loss/FRt4/detection_loss‚3.2250489755110308 1loss/SRqqt1/detection_loss‚2.1106827280351093 1loss/SRqqt2/detection_loss‚2.2319770073890686 &loss/total_loss‚41.078878676126166 /loss/FRt1/detection_loss‚2.2385232216935367 /loss/FRt3/assignment_loss‚1.473046606495267‘ªÇX¯““#‚(:ˆ ¡ÚéÔ‡­’n gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚120.927' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.23175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²D‘ør”#‚(:ç ¡ÚéÔàÃo disk.sda1.in‚0 disk.sdb1.in‚0 cpu‚25.185636535227477 disk.sda2.in‚8.515625" disk.sda2.out‚1051.32568359375 disk.rbd2.in‚0! proc.memory.rssMB‚ 3004.578125 network.recv‚ 3.064013e+06" disk.nvme0n1.out‚ 103142.140625# disk.nvme1n1.out‚38971.83984375$ memory_percent‚91.95168018341064 disk.sdb2.in‚3.78125 disk.sdb1.out‚0) proc.memory.availableMB‚ 1318.63671875* proc.memory.percent‚18.338489532470703) disk./.usagePercent‚75.77685537275552" disk.sdb2.out‚1051.32568359375 disk.sda1.out‚0! disk.sdb.out‚1051.32568359375 disk.nvme1n1.in‚ 744.32421875 network.sent‚ 2.076337e+07 disk.rbd0.out‚ 2.466796875 disk.nvme0n1.in‚ 939.5390625 disk.md0.in‚ 12.296875 disk.rbd2.out‚ 2.69140625 disk.rbd0.in‚0! disk.dm-0.out‚142113.98046875 proc.cpu.threads‚17 disk.md0p1.in‚ 12.296875 disk.md0p1.out‚ 1011.37109375 disk.dm-0.in‚ 1683.86328125 disk.sdb.in‚3.78125 disk.md0.out‚ 1011.37109375$ disk./.usageGB‚2710.147392272949 disk.sda.in‚8.515625! disk.sda.out‚1051.32568359375æäÁ‘•#‚(:† °ÚéÔõò•m gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.988' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.747 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³ ù%Ò–#‚(:â °ÚéÔ‡ÔÍm* proc.memory.percent‚18.338489532470703 disk.nvme1n1.in‚ 744.32421875 disk.sda1.out‚0 disk.md0p1.in‚ 12.296875 disk.dm-0.in‚ 1683.86328125 disk.sdb2.in‚3.78125 network.recv‚ 3.06944e+06 disk.rbd2.in‚0 disk.sdb1.out‚0 cpu‚25.185403246339572 disk.nvme0n1.in‚ 939.5390625 disk.rbd0.in‚0 disk.rbd0.out‚ 2.466796875 disk.md0p1.out‚ 1013.07421875 disk.dm-0.out‚ 142143.046875 di ¦Ut›sk.sda2.in‚8.515625 disk.sdb.in‚3.78125 disk.nvme1n1.out‚ 38972.03125 disk.md0.out‚ 1013.07421875 disk.rbd2.out‚ 2.69140625 disk.sdb1.in‚0" disk.sda2.out‚1053.09716796875! disk.sda.out‚1053.09716796875) proc.memory.availableMB‚ 1318.86328125" disk.sdb2.out‚1053.09716796875! proc.memory.rssMB‚ 3004.578125 proc.cpu.threads‚17) disk./.usagePercent‚75.78035522479603$ disk./.usageGB‚2710.272563934326 disk.sda1.in‚0$ memory_percent‚91.95029735565186! disk.sdb.out‚1053.09716796875 network.sent‚ 2.0870824e+07" disk.nvme0n1.out‚ 103171.015625 disk.md0.in‚ 12.296875 disk.sda.in‚8.515625Ц•·‘—#‚(:† ¿ÚéÔåêm gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚120.65' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.1625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z4M€g˜#‚(:Ü ¿ÚéÔ®šn disk.sda.out‚1055.001953125 disk.sda1.out‚0 disk.sdb2.in‚3.78125 disk.sdb2.out‚1055.001953125 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0" disk.nvme0n1.out‚ 103438.609375 disk.nvme1n1.in‚ 744.32421875 disk.md0.in‚ 12.296875 disk.md0p1.in‚ 12.296875! disk.dm-0.out‚142411.02734375 disk.md0.out‚ 1014.9140625 disk.rbd2.out‚ 2.69140625$ memory_percent‚91.95189476013184! proc.memory.rssMB‚ 3004.578125$ disk./.usageGB‚2710.397689819336 disk.sdb.in‚3.78125 disk.rbd0.out‚ 2.466796875 disk.md0p1.out‚ 1014.9140625 disk.sda.in‚8.515625 proc.cpu.threads‚17 disk.nvme0n1.in‚ 939.5390625# disk.nvme1n1.out‚38972.41796875 disk.rbd2.in‚0 disk.sda1.in‚0* proc.memory.percent‚18.338489532470703 disk.sdb.out‚1055.001953125 disk.sda2.in‚8.515625 disk.sda2.out‚1055.001953125( proc.memory.availableMB‚ 1318.6015625 cpu‚25.18513322590189 network.sent‚ 2.0882123e+07 network.recv‚ 3.080121e+06) disk./.usagePercent‚75.78385379691015 disk.dm-0.in‚ 1683.86328125q'Ëä“™#‚(:ˆ ÎÚéÔý¥µm gpu.0.gpu‚34.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚117.525' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.38125 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MÁ†çeš#‚(:Ú ÎÚéÔìçn disk.sdb.out‚1056.509765625% disk./.usageGB‚2710.6483116149902 disk.sdb1.out‚0 disk.sdb.in‚3.78125 network.sent‚ 2.0888398e+07 network.recv‚ 3.084948e+06 disk.md0p1.out‚ 1016.34765625 disk.dm-0.out‚142803.6953125 disk.md0.in‚ 12.296875 disk.rbd2.out‚2.765625$ memory_percent‚91.95201396942139* proc.memory.percent‚18.338489532470703 disk.rbd0.in‚0 disk.nvme0n1.in‚ 939.5390625 disk.nvme1n1.out‚ 38972.625 disk.sdb2.in‚3.78125 disk.sda.out‚1056.509765625 disk.sda1.out‚0 cpu‚25.184893586848645 proc.cpu.threads‚17 disk.dm-0.in‚ 1683.86328125 disk.nvme1n1.in‚ 744.32421875 disk.sdb2.out‚1056.509765625 disk.sda1.in‚0! proc.memory.rssMB‚ 3004.578125 disk.md0p1.in‚ 12.296875 disk.md0.out‚ 1016.34765625 disk.rbd2.in‚0 disk.sda2.in‚8.515625# disk.nvme0n1.out‚103831.0703125) disk./.usagePercent‚75.79086128721002 disk.rbd0.out‚ 2.478515625 disk.sdb1.in‚0 disk.sda2.out‚1056.509765625 disk.sda.in‚8.515625) proc.memory.availableMB‚ 1318.58203125[ŠØ2›#- _wandbruntime‚9482  _runtime‚9482»ÿ…Éœ#à ' lr-AdamW/pg1‚0.00031930434727586395 ' lr-AdamW/pg2‚0.00031930434727586395 trainer/global_step‚8049 % _timestamp‚1.7885054243048697e+09 _runtime‚9482.082177952 _step‚658žÿIgÎ#È ' lr-AdamW/pg2‚0.00031930434727586395 trainer/global_step‚8049 % _timestamp‚1.7885054243048697e+09 _runtime‚9482.082177952 _step‚658 ' lr-AdamW/pg1‚0.00031930434727586395’…Â!2ž#- _wandbruntime‚9482  _runtime‚9482šº‰þ'Ÿ#øO UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100744734 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚677568102531584 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9623636511232 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9180517319602 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚659 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9214974106112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚629 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚747607070517760 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚677568120534528 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚747607046577664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚677568102531584 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 203788369 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 149489914 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48745180 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚686783076637696 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚631 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚664355980788542 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚664355055774178 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚9482.082967266 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 % _timestamp‚1.7885054243056273e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 371937091 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48745809 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 168148722 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 203762025 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9623405632000 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 168148720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚757230707028992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 203788369 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚757230452209664 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚655174538454576 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚686783076637696 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9215884077568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9181425294606 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 168148720 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 149490545 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144”ïˆ( #ýO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9214974106112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚629 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚747607070517760 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚677568120534528 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚747607046577664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚677568102531584 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 203788369 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 149489914 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48745180 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚686783076637696 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚631 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚664355980788542 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚664355055774178 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚9482.082967266 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 % _timestamp‚1.7885054243056273e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 371937091 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48745809 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 168148722 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 203762025 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9623405632000 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 168148720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚757230707028992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 203788369 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚757230452209664 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚655174538454576 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚686783076637696 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9215884077568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9181425294606 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 168148720 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 149490545 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100744734 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚677568102531584 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9623636511232 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9180517319602 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚659“v}Ü2¡#- _wandbruntime‚9483  _runtime‚9483êlì¢#‚N ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚655253556049120 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚677649739620352 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚686866745171456 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚757322102848000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 203813226 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9217005551104 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 168168880 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚677649721617408 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100756799 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 371955736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149507752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚664436115656572 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚664435190642000 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚677649721617408 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9216095566336 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9624806625792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168168880 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 168168882 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚626 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9624579954176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚660 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9182542568092 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 371982108 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 203786856 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168168882 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚655253573088480 JBDeviceStatsMonitor.on_train_batch_end/segment.à+ìlarge_pool.allocated‚615 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚757321852236288 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 371982108 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚686865817183744 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 203786856 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9216095566336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚747697296222208 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚677649739620352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 371955736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚747697272282112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 % _timestamp‚1.7885054254338176e+09gfô '£#‡N JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚757321852236288 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 371982108 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚686865817183744 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 203786856 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9216095566336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚747697296222208 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚677649739620352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 371955736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚747697272282112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 % _timestamp‚1.7885054254338176e+09 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚655253556049120 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚677649739620352 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚686866745171456 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚757322102848000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 203813226 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9217005551104 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 168168880 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚677649721617408 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100756799 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 371955736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149507752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚664436115656572 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚664435190642000 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚677649721617408 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9216095566336 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9624806625792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168168880 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 168168882 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚626 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9624579954176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚660 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9182542568092 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 371982108 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 203786856 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168168882 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚655253573088480”êžÖ2¤#- _wandbruntime‚9483  _runtime‚9483Y×%Š[¥#Õ 1loss/SRqqt3/assignment_loss‚0.988537928014172 1loss/FBt3/assignment_loss‚0.07411379293334903 /loss/FRt4/detection_loss‚2.7903011957804362 1loss/FBt1/assignment_loss‚0.11538079844666609 epoch‚6 % _timestamp‚1.7885054254370139e+09 _runtime‚9483.214296987 .loss/FRt3/detection_loss‚1.503251067052285 _step‚661 1loss/FBt2/assignment_loss‚0.14468713874214245 /loss/FRt1/detection_loss‚2.0851405390065336 %loss/total_loss‚41.83917262020141 /loss/FRt3/assignment_loss‚1.342807721007954 1loss/SRqqt4/detection_loss‚2.1485480964183807 trainer/global_step‚8049 0loss/FRt2/assignment_loss‚2.8940313895543417 2loss/SRqqt2/assignment_loss‚1.7730042103897123 /loss/FBt1/detection_loss‚0.9255861220780112 /loss/FRt2/detection_loss‚2.3183175106843312 0loss/SRqqt1/detection_loss‚2.211353726685047 0loss/SRqqt2/detection_loss‚2.465239868444555 1loss/SRqqt3/detection_loss‚0.7749347648419002 /loss/FBt3/detection_loss‚2.3464339077472687 /loss/FRt4/assignment_loss‚2.892570638656616 1loss/SRqqt4/assignment_loss‚1.690591722726822 1loss/FBt4/assignment_loss‚0.22746297111734748 .loss/FBt2/detection_loss‚2.542361701528231 /loss/FBt4/detection_loss‚3.9458547915731157 0loss/FRt1/assignment_loss‚2.4031368777155877 2loss/SRqqt1/assignment_loss‚1.2355241390566032>öVr`¦#Ú 1loss/FBt2/assignment_loss‚0.14468713874214245 /loss/FRt1/detection_loss‚2.0851405390065336 %loss/total_loss‚41.83917262020141 /loss/FRt3/assignment_loss‚1.342807721007954 1loss/SRqqt4/detection_loss‚2.1485480964183807 trainer/global_step‚8049 0loss/FRt2/assignment_loss‚2.8940313895543417 2loss/SRqqt2/assignment_loss‚1.7730042103897123 /loss/FBt1/detection_loss‚0.9255861220780112 /loss/FRt2/detection_loss‚2.3183175106843312 0loss/SRqqt1/detection_loss‚2.211353726685047 0loss/SRqqt2/detection_loss‚2.465239868444555 1loss/SRqqt3/detection_loss‚0.7749347648419002 /loss/FBt3/detection_loss‚2.3464339077472687 /loss/FRt4/assignment_loss‚2.892570638656616 1loss/SRqqt4/assignment_loss‚1.690591722726822 1loss/FBt4/assignment_loss‚0.22746297111734748 .loss/FBt2/detection_loss‚2.542361701528231 /loss/FBt4/detection_loss‚3.9458547915731157 0loss/FRt1/assignment_loss‚2.4031368777155877 2loss/SRqqt1/assignment_loss‚1.2355241390566032 1loss/SRqqt3/assignment_loss‚0.988537928014172 1loss/FBt3/assignment_loss‚0.07411379293334903 /loss/FRt4/detection_loss‚2.7903011957804362 1loss/FBt1/assignment_loss‚0.11538079844666609 epoch‚6 % _timestamp‚1.7885054254370139e+09 _runtime‚9483.214296987 .loss/FRt3/detection_loss‚1.503251067052285 _step‚661•Ñžvl§#‚(:á ÝÚéÔñÝÁm disk.sda1.in‚0 disk.nvme0n1.in‚ 939.5390625* proc.memory.percent‚18.338489532470703! proc.memory.rssMB‚ 3004.578125 proc.cpu.threads‚17$ disk./.usageGB‚2710.648349761963 disk.sda2.in‚8.515625 disk.sda.in‚8.515625 disk.sda1.out‚0 disk.rbd0.out‚ 2.541015625 disk.sdb1.in‚0 disk.dm-0.in‚ 1683.86328125! disk.nvme1n1.out‚ 38972.890625 disk.md0p1.in‚ 12.296875 cpu‚25.184684945661775 network.sent‚ 2.0904295e+07) disk./.usagePercent‚75.79086235381534 disk.sdb.in‚3.78125 disk.nvme1n1.in‚ 744.32421875 disk.sda.out‚1058.1435546875 disk.md0p1.out‚ 1017.91015625! disk.dm-0.out‚142975.07421875( proc.memory.availableMB‚ 1318.2890625 network.recv‚ 3.087964e+06 disk.sdb.out‚1058.1435546875 disk.md0.in‚ 12.296875 disk.rbd0.in‚0 disk.sdb1.out‚0$ memory_percent‚91.95380210876465! disk.sda2.out‚1058.1435546875$ disk.nvme0n1.out‚104002.18359375 disk.md0.out‚ 1017.91015625 disk.rbd2.in‚0 disk.rbd2.out‚2.765625 disk.sdb2.in‚3.78125! disk.sdb2.out‚1058.1435546875Ììà¨#‚(:’ ÝÚéÔÂä§n gpu.0.gpu‚18.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚127.106' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚31.776499999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Óýôœ©#‚(:‘ ìÚéÔÎêŠl gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚117.81' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.452499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›K¹rª#‚(:ç ìÚéÔÑÞ‡n disk.rbd2.out‚ 2.86767578125 disk.sdb2.in‚3.78125! proc.memory.rssMB‚ 3004.578125 disk.rbd0.in‚0 disk.md0p1.in‚ 12.296875! disk.dm-0.out‚143009.80859375! disk.nvme0n1.out‚ 104036.71875 disk.rbd2.in‚0 disk.sda2.in‚8.515625# disk.nvme1n1.out‚38973.08984375) proc.memory.availableMB‚ 1318.76171875 disk.md0p1.out‚ 1019.6484375 disk.dm-0.in‚ 1683.86328125! disk.sdb.out‚1059.94482421875 disk.nvme0n1.in‚ 939.5390625 cpu‚25.18444396288722( disk./.usagePercent‚75.7907861981954! disk.sda.out‚1059.94482421875 disk.sdb1.out‚0* proc.memory.percent‚18.338489532470703 network.sent‚ 2.1001468e+07" disk.sdb2.out‚1059.94482421875 disk.sda.in‚8.515625 disk.sda1.out‚0 disk.sdb1.in‚0 disk.sdb.in‚3.78125 network.recv‚ 3.093534e+06 disk.sda1.in‚0 disk.rbd0.out‚ 2.541015625" disk.sda2.out‚1059.94482421875 disk.nvme1n1.in‚ 744.32421875 disk.md0.in‚ 12.296875 disk.md0.out‚ 1019.6484375 proc.cpu.threads‚17% disk./.usageGB‚2710.6456260681152$ memory_percent‚91.95091724395752c’¯¼«#‚(:… ûÚéÔèËm gpu.0.gpu‚9.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.264' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.816 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«#áÏ`¬#‚(:Õ ûÚéÔ¸‹³n disk.sda.in‚8.515625 disk.md0p1.out‚ 1021.17578125 disk.sdb.out‚ 1061.53515625 disk.sdb1.in‚0 disk.nvme1n1.out‚ 38973.6875 disk.md0.out‚ 1021.17578125 disk.rbd2.in‚0$ memory_percent‚91.95225238800049 disk.sda1.in‚0 disk.sdb.in‚3.78125! proc.memory.rssMB‚ 3004.578125 proc.cpu.threads‚17% disk./.usageGB‚2710.6456260681152 disk.sda.out‚ 1061.53515625 disk.sdb1.out‚0 disk.rbd2.out‚ 2.86767578125) proc.memory.availableMB‚ 1318.54296875( disk./.usagePercent‚75.7907861981954 disk.sdb2.out‚ 1061.53515625 disk.sda1.out‚0 disk.dm-0.out‚ 143220.375 disk.nvme1n1.in‚ 744.32421875 network.recv‚ 3.097596e+06 disk.rbd0.in‚0 disk.rbd0.out‚ 2.6201171875 disk.dm-0.in‚ 1683.86328125 disk.sda2.in‚8.515625* proc.memory.percent‚18.338489532470703 disk.sdb2.in‚3.78125 disk.md0p1.in‚ 12.296875 disk.sda2.out‚ 1061.53515625 disk.nvme0n1.in‚ 939.5390625 network.sent‚ 2.1007297e+07 disk.nvme0n1.out‚ 104246.6875 disk.md0.in‚ 12.296875 cpu‚25.184204653086876nÉ[12­#- _wandbruntime‚9539  _runtime‚9539Õ ÕWÉ®#à ' lr-AdamW/pg1‚0.00031234189061727193 ' lr-AdamW/pg2‚0.00031234189061727193 trainer/global_step‚8099 % _timestamp‚1.7885054818976448e+09 _runtime‚9539.675029314 _step‚662/"`¦Î¯#È _step‚662 ' lr-AdamW/pg1‚0.00031234189061727193 ' lr-AdamW/pg2‚0.00031234189061727193 trainer/global_step‚8099 % _timestamp‚1.7885054818976448e+09 _runtime‚9539.675029314–‘á¨×2°#- _wandbruntime‚9539  _runtime‚9539â)…{þ'±#øO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 205006927 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 205033271 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9682326663680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚690920118353920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚761800431778816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49035944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9236394574710 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚8099 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚690921046328320 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚668361812756486 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚659125418181776 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚690920118353920 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 374163647 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 169156720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150383930 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 169156720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49035310 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 374189993 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 169156722 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9271971502592 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7885054818992276e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101347986 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9271061531136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 _step‚663 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 169156722 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚681649056822784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 150383294 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚690921046328320 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚668362737770850¬Yds ²#ýO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9271061531136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 _step‚663 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 169156722 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚681649056822784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 150383294 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚690921046328320 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚668362737770850 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 205006927 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 MGDeviceStatsMonito)D&Ðr.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 205033271 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9682326663680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚690920118353920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚761800431778816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49035944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9236394574710 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚8099 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚690921046328320 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚668361812756486 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚659125418181776 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚690920118353920 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 374163647 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 169156720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150383930 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 169156720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49035310 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 374189993 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 169156722 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9271971502592 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7885054818992276e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101347986 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0—^½Š“³#‚(:ˆ ŠÛéÔ«º n gpu.0.gpu‚17.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.327' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.83175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g†˜ùv´#‚(:ë ŠÛéÔÙìõn disk.rbd2.in‚0 disk.rbd2.out‚ 2.9072265625 disk.rbd0.in‚0! disk.sdb.out‚1062.94091796875 disk.nvme0n1.in‚ 939.5390625 disk.nvme1n1.in‚ 744.32421875 disk.md0p1.out‚ 1022.51171875 network.recv‚ 3.102277e+06" disk.sdb2.out‚1062.94091796875 disk.sda.in‚8.515625! disk.sda.out‚1062.94091796875 disk.sda1.in‚0) proc.memory.availableMB‚ 1317.94140625 network.sent‚ 2.1011414e+07* proc.memory.percent‚18.338489532470703 disk.sdb.in‚3.78125 disk.sda2.in‚8.515625 disk.md0.in‚ 12.296875 disk.sdb1.in‚0 disk.dm-0.in‚ 1683.86328125# disk.nvme1n1.out‚38973.85546875 disk.md0.out‚ 1022.51171875$ disk./.usageGB‚2710.647605895996" disk.sda2.out‚1062.94091796875 disk.dm-0.out‚ 143718.671875 disk.sdb2.in‚3.78125 disk.sda1.out‚0 disk.rbd0.out‚ 2.6201171875 proc.cpu.threads‚17) disk./.usagePercent‚75.79084155501158$ disk.nvme0n1.out‚104744.81640625 disk.sdb1.out‚0 disk.md0p1.in‚ 12.296875$ memory_percent‚91.95592403411865! proc.memory.rssMB‚ 3004.578125 cpu‚25.183965198067547Â=ů2µ#- _wandbruntime‚9540  _runtime‚9540ÕJÓ5'¶#N OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 205058128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 trainer/global_step‚8099 _runtime‚9540.717949295 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9273092976128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 245545984 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚761891802893824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 269486080 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 169176880 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49041654 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 374208638 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚752208581444608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150401705 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _step‚664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9273092976128 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚681730693911552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚659204452815680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚659204435776320 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚668442872638880 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9272182991360 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 374235010 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49040993 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150401042 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9683245389312 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9237511847988 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 205031758 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚691002858899968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9683490935296 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚668441947624308 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 169176882 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚681730675908608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 169176882 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 169176880 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0@S]| '·#†N XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 269486080 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 169176880 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49041654 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 374208638 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚752208581444608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150401705 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _step‚664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9273092976128 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚681730693911552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚659204452815680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚659204435776320 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚668442872638880 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9272182991360 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 374235010 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49040993 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150401042 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9683245389312 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9237511847988 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 205031758 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚691002858899968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9683490935296 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚668441947624308 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 169176882 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚681730675908608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 169176882 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 169176880 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 205058128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 trainer/global_step‚8099 _runtime‚9540.717949295 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9273092976128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 245545984 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚761891802893824˜@ɉÑ2¸#- _wandbruntime‚9540  _runtime‚95408 lÅZ¹#Ô /loss/FBt4/detection_loss‚2.4839448511600493 #loss/total_loss‚39.981672845189 .loss/FRt2/detection_loss‚2.084543654024601 0loss/FRt3/assignment_loss‚1.3871838758389154 /loss/FRt4/assignment_loss‚2.634042744835218 /loss/FRt4/detection_loss‚2.8020474116007485 _step‚665 /loss/FRt1/assignment_loss‚2.754898172285822 /loss/FRt3/detection_loss‚1.5101830477205416 1loss/SRqqt1/detection_loss‚1.9611105929160941 .loss/FBt3/detection_loss‚2.584252781338162 1loss/SRqqt4/assignment_loss‚1.859473590180278 2loss/FBt2/assignment_loss‚0.002568645615656351 1loss/SRqqt2/detection_loss‚2.2659259513020515 % _timestamp‚1.7885054829425793e+09 2loss/SRqqt1/assignment_loss‚1.3391493309674591 2loss/SRqqt2/assignment_loss‚1.4805644960142672 /loss/FRt1/detection_loss‚2.0275384171311797 epoch‚6 0loss/FRt2/assignment_loss‚2.7137941694259644 1loss/FBt1/assignment_loss‚0.14747557390785307 1loss/FBt4/assignment_loss‚0.10657104659985635 1loss/SRqqt3/detection_loss‚0.9960365050250576 _runtime‚9540.719592982 .loss/FBt2/detection_loss‚2.623692238330841 1loss/FBt3/assignment_loss‚0.16434990187270967 trainer/global_step‚8099 1loss/SRqqt3/assignment_loss‚1.051384256385444 0loss/SRqqt4/detection_loss‚2.459776218732198 /loss/FBt1/detection_loss‚0.5411653719780346[ä%>_º#Ù epoch‚6 0loss/FRt2/assignment_loss‚2.7137941694259644 1loss/FBt1/assignment_loss‚0.14747557390785307 1loss/FBt4/assignment_loss‚0.10657104659985635 1loss/SRqqt3/detection_loss‚0.9960365050250576 _runtime‚9540.719592982 .loss/FBt2/detection_loss‚2.623692238330841 1loss/FBt3/assignment_loss‚0.16434990187270967 trainer/global_step‚8099 1loss/SRqqt3/assignment_loss‚1.051384256385444 0loss/SRqqt4/detection_loss‚2.459776218732198 /loss/FBt1/detection_loss‚0.5411653719780346 /loss/FBt4/detection_loss‚2.4839448511600493 #loss/total_loss‚39.981672845189 .loss/FRt2/detection_loss‚2.084543654024601 0loss/FRt3/assignment_loss‚1.3871838758389154 /loss/FRt4/assignment_loss‚2.634042744835218 /loss/FRt4/detection_loss‚2.8020474116007485 _step‚665 /loss/FRt1/assignment_loss‚2.754898172285822 /loss/FRt3/detection_loss‚1.5101830477205416 1loss/SRqqt1/detection_loss‚1.9611105929160941 .loss/FBt3/detection_loss‚2.584252781338162 1loss/SRqqt4/assignment_loss‚1.859473590180278 2loss/FBt2/assignment_loss‚0.002568645615656351 1loss/SRqqt2/detection_loss‚2.2659259513020515 % _timestamp‚1.7885054829425793e+09 2loss/SRqqt1/assignment_loss‚1.3391493309674591 2loss/SRqqt2/assignment_loss‚1.4805644960142672 /loss/FRt1/detection_loss‚2.0275384171311797™UcR“»#‚(:ˆ ™ÛéÔÚË‚n gpu.0.gpu‚15.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚184.199' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚46.04975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\ƒAW¼#‚(:ä ™ÛéÔŒ¯½n disk.dm-0.in‚ 1683.86328125 cpu‚25.183731736422306" @, disk.nvme1n1.out‚ 38974.1328125 disk.dm-0.out‚ 143876.46875 disk.nvme0n1.in‚ 939.5390625# disk.nvme0n1.out‚104902.3359375 disk.rbd2.in‚0$ memory_percent‚91.95597171783447* proc.memory.percent‚18.338489532470703 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.md0p1.in‚ 12.296875 disk.md0p1.out‚ 1024.15625 disk.sda2.in‚8.515625" disk.sda2.out‚1064.65771484375 disk.nvme1n1.in‚ 744.32421875) disk./.usagePercent‚75.77609946956437! disk.sda.out‚1064.65771484375 disk.md0.out‚ 1024.15625 disk.sdb2.in‚3.78125" disk.sdb2.out‚1064.65771484375 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚ 2.68310546875 disk.sdb1.in‚0 disk.sdb.in‚3.78125 disk.rbd2.out‚ 2.9072265625) proc.memory.availableMB‚ 1317.93359375 disk.md0.in‚ 12.296875 network.recv‚ 3.106074e+06% disk./.usageGB‚2710.1203575134277! proc.memory.rssMB‚ 3004.578125 disk.sda.in‚8.515625! disk.sdb.out‚1064.65771484375 network.sent‚ 2.1119809e+07 disk.sdb1.out‚0}2Se“½#‚(:ˆ ¨ÛéÔõ±‚m gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚148.223' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.05575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©%dr¾#‚(:ç ¨ÛéÔù§’n disk.sda2.in‚8.515625 disk.nvme1n1.in‚ 744.32421875 network.sent‚ 2.1205589e+07 network.recv‚ 3.111051e+06) disk./.usagePercent‚75.77608869685064 disk.dm-0.out‚ 144134.984375 disk.sdb.out‚1066.6376953125# disk.nvme1n1.out‚38974.35546875 disk.md0.in‚ 12.296875 disk.rbd2.in‚0$ disk.nvme0n1.out‚105160.62890625 disk.sda.out‚1066.6376953125 disk.rbd0.in‚0 disk.sdb.in‚3.78125 disk.nvme0n1.in‚ 939.5390625 disk.sda1.out‚0 disk.sdb1.out‚0 disk.md0.out‚ 1026.0703125! disk.sdb2.out‚1066.6376953125! proc.memory.rssMB‚ 3004.578125 proc.cpu.threads‚17 disk.sda1.in‚0 cpu‚25.183498495145198 disk.sdb1.in‚0 disk.md0p1.out‚ 1026.0703125 disk.dm-0.in‚ 1683.86328125! disk.sda2.out‚1066.6376953125 disk.rbd2.out‚ 2.97021484375 disk.sdb2.in‚3.78125* proc.memory.percent‚18.338489532470703$ disk./.usageGB‚2710.119972229004 disk.md0p1.in‚ 12.296875 disk.sda.in‚8.515625 disk.rbd0.out‚ 2.68310546875$ memory_percent‚91.95616245269775) proc.memory.availableMB‚ 1317.90234375d ²“¿#‚(:ˆ ·ÛéÔªÍòl gpu.0.gpu‚52.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚117.525' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.38125 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hfÉtÀ#‚(:é ·ÛéÔâŽÚm disk.sda1.in‚0 cpu‚25.183266518962718) disk./.usagePercent‚75.77680694887394% disk./.usageGB‚2710.1456604003906 disk.sdb1.out‚0 disk.md0p1.out‚ 1027.70703125 disk.nvme0n1.in‚ 939.5390625) proc.memory.availableMB‚ 1318.15234375 disk.sdb2.in‚3.78125" disk.sdb2.out‚1068.34423828125! disk.sda.out‚1068.34423828125 disk.sda1.out‚0$ disk.nvme0n1.out‚105161.37109375 network.sent‚ 2.1211448e+07 network.recv‚ 3.115135e+06 disk.rbd0.in‚0! proc.memory.rssMB‚ 3004.578125* proc.memory.percent‚18.338489532470703 disk.rbd0.out‚ 2.74609375! disk.sdb.out‚1068.34423828125 disk.sda2.in‚8.515625 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.nvme1n1.in‚ 744.32421875 disk.nvme1n1.out‚ 38974.71875 disk.md0.in‚ 12.296875 disk.sda.in‚8.515625 disk.sdb1.in‚0 disk.md0p1.in‚ 12.296875 disk.sdb.in‚3.78125 disk.dm-0.in‚ 1683.86328125" disk.sda2.out‚1068.34423828125 disk.rbd2.out‚ 2.97021484375# memory_percent‚91.9546365737915! disk.dm-0.out‚144136.08984375 disk.md0.out‚ 1027.70703125’Q´à2Á#- _wandbruntime‚9597  _runtime‚9597ñèÉÂ#à ' lr-AdamW/pg1‚0.00030542399684739817 ' lr-AdamW/pg2‚0.00030542399684739817 trainer/global_step‚8149 % _timestamp‚1.7885055394473004e+09 _runtime‚9597.224826047 _step‚6666½‰ÎÃ#È ' lr-AdamW/pg1‚0.00030542399684739817 ' lr-AdamW/pg2‚0.00030542399684739817 trainer/global_step‚8149 % _timestamp‚1.7885055394473004e+09 _runtime‚9597.224826047 _step‚666šÈ?2Ä#- _wandbruntime‚9597  _runtime‚9597Œ„Öþ'Å#øO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚672368569738794 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 376416549 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚756629640962560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚756629617022464 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚663076314948336 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚663076297908976 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 170164720 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚695058088044544 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 170164722 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚695057160070144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚685730011113984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 376442895 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9328058927616 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9293179804822 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚685730029116928 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 101951234 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9327148956160 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206251829 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151276830 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚766370251261440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 170164722 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9740634238976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 _step‚667 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 170164720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 206278173 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9327148956160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚633 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚766370506080768 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 206278173 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101951236 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49325594 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9740865118208 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 206251829 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 trainer/global_step‚8149 _runtime‚9597.226365454 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9292271829818 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚695057160070144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49324963 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0ýë(Æ#ýO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9327148956160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚633 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚766370506080768 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 206278173 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101951236 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49325594 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9740865118208 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 206251829 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 trainer/global_step‚8149 _runtime‚9597.226365454 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9292271829818 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚695057160070144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49324963 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚672368569738794 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 376416549 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚756629640962560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚756629617022464 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚663076314948336 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚663076297908976 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 170164720 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚695058088044544 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 170164722 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚695057160070144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚685730011113984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 376442895 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9328058927616 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9293179804822 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚685730029116928 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 101951234 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9327148956160 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206251829 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151276830 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚766370251261440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 170164722 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9740634238976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 _step‚667 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 170164720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 206278173 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0›ÊÒ¦™2Ç#- _wandbruntime‚9598  _runtime‚9598Â9†ÅSÈ#‚N N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 376487912 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9328270416384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9329180401152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9742034880512 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885055405311384e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170184882 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚663155315503520 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 170184880 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚685811630199808 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚695140828603904 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 170184880 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9741814500352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚8149 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 206276660 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚685811648202752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚631 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49330738 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101963299 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 206276660 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 _step‚668 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 206303030 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚766461901547520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9329180401152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚672449629621188 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚672448704606616 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚663155332542880 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚766461657227264 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceS´‘-»µ tatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9328270416384 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 376461540 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 151294670 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151294037 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101963301 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 376461540 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚695139900616192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244320256 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49331369 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚695140828603904 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚756719866667008 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚756719842726912 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9294297078308 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9293389103096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220380160 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2-ƒæy 'É#‡N L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 206276660 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 _step‚668 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 206303030 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚766461901547520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9329180401152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚672449629621188 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚672448704606616 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚663155332542880 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚766461657227264 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9328270416384 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 376461540 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 151294670 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151294037 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101963301 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 376461540 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚695139900616192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244320256 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49331369 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚695140828603904 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚756719866667008 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚756719842726912 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9294297078308 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9293389103096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220380160 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 376487912 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9328270416384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9329180401152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9742034880512 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885055405311384e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170184882 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚663155315503520 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 170184880 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚685811630199808 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚695140828603904 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 170184880 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9741814500352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚8149 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 206276660 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚685811648202752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚631 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49330738 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101963299 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚699195129760768 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152169585 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚770940272181760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237170688 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 207496731 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚667027194675536 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚761140926184960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9799108826112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚8199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9384146352640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚699194201786368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9349057059930 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚689810983408128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 171172720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚699195129760768 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚667027177636176 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚699194201786368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9384146352640 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49615760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 % _timestamp‚1.7885055970964396e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚689810983408128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚689810965405184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49615101 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 207523075 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0šaÁRçÚ#ýO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49615760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 % _timestamp‚1.7885055970964396e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚689810983408128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚689810965405184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49615101 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 207523075 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 152170246 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚659 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚676376251735466 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 _runtime‚9654.873744251 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚689810965405184 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚699195129760768 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152169585 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚770940272181760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237170688 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 207496731 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚667027194675536 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚761140926184960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9799108826112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚8199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9384146352640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚699194201786368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9349057059930 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚689810983408128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 171172720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚699195129760768 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚667027177636176 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚699194201786368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9384146352640Ÿîþm˜2Û#- _wandbruntime‚9655  _runtime‚9655·/¥j'Ü#‚N I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 207521562 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9384357841408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9384357841408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 378740814 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 207521562 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 171192880 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9385267826176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚689892584491008 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚699277870320128 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 171192882 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49620916 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚771031663190016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152187465 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 207547932 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚689892602493952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 trainer/global_step‚8199 _runtime‚9655.984076573 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚699276942332416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9800282531840 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚667106195230720 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚676455461588924 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49621554 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9349266358204 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _step‚672 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 378714442 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 207547932 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚761231127949312 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚689892584491008 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚771031410481152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 171192882 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 152188105 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144®¾@] 'Ý#‡N GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 102566549 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 207547932 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚761231127949312 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚689892584491008 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚771031410481152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 171192882 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 152188105 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 207521562 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9384357841408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9384357841408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 378740814 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 207521562 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 171192880 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9385267826176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚689892584491008 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚699277870320128 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 171192882 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49620916 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚771031663190016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152187465 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 207547932 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚689892602493952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 trainer/global_step‚8199 _runtime‚9655.984076573 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚699276942332416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9800282531840 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚667106195230720 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚676455461588924 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49621554 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9349266358204 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _step‚672 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 378714442 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 % _timestamp‚1.7885056547652307e+09 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 103157734 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚703332171476992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _runtime‚9712.542917341 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9439323806208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153063913 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚8249 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚703332171476992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 380922353 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚703331243502592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 172180722 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9439323806208 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9858045264384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚693891919696384 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103157736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242236416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 380948699 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0TDí(î#ýO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚703331243502592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 172180722 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9439323806208 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9858045264384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚693891919696384 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103157736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242236416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 380948699 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 208767977 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 % _timestamp‚1.7885056547652307e+09 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 103157734 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚703332171476992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _runtime‚9712.542917341 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9439323806208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153063913 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚8249 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚703332171476992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 380922353£Ý·2ï#- _wandbruntime‚9713  _runtime‚9713ü"&æð#N XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9440445266432 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 172200882 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153081690 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49911245 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 172200880 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚703413984048640 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚765742413171712 _runtime‚9713.614220341 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚680462218571232 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚693973556785152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9440445266432 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚644 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9859002189312 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 208792834 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 380967344 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9406051588524 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚680463143585804 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 $ _timestamp‚1.788505655836314e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 380993716 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 103169799 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚775601415361024 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 208766464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9859228860928 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 172200882 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 380967344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9441355251200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9441355251200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 _step‚676 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚703414912036352 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 208766464 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚671057091997280 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D;DeviceStatsM|½Ì!onitor.on_train_batch_end/segment.all.allocated‚1185 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚646 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 103169801 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153081044Ô¥´t 'ñ#†N K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 208766464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9859228860928 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 172200882 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 380967344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9441355251200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9441355251200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 _step‚676 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚703414912036352 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 208766464 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚671057091997280 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚646 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 103169801 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153081044 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9440445266432 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 172200882 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153081690 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49911245 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 172200880 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚703413984048640 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚765742413171712 _runtime‚9713.614220341 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚680462218571232 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚693973556785152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9440445266432 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚644 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9859002189312 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 208792834 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 380967344 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9406051588524 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚680463143585804 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 $ _timestamp‚1.788505655836314e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 380993716 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 103169799 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚775601415361024 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570¤ðõãQ2ò#- _wandbruntime‚9713  _runtime‚9713RÒ–Yó#Ó 1loss/FBt2/assignment_loss‚0.16313559784170398 /loss/FBt1/detection_loss‚0.7537185139281832 .loss/FBt2/detection_loss‚2.729530772992543 2loss/SRqqt1/assignment_loss‚1.1853086085369189 /loss/FBt3/detection_loss‚3.0339554250240326 .loss/FBt4/detection_loss‚2.142945820635015 epoch‚6 trainer/global_step‚8249 % _timestamp‚1.7885056558388312e+09 /loss/FRt2/assignment_loss‚2.901768916032531 1loss/FBt1/assignment_loss‚0.12025907197396239 -loss/FRt2/detection_loss‚2.18857678711753 0loss/FBt4/assignment_loss‚0.5021955508032182 1loss/SRqqt4/detection_loss‚2.5820091466108956 2loss/SRqqt3/assignment_loss‚1.1088233074808325 2loss/SRqqt4/assignment_loss‚1.9770086467266084 /loss/FRt4/detection_loss‚3.1640046644210815 0loss/SRqqt1/detection_loss‚1.678294860240486 _step‚677 0loss/FRt1/assignment_loss‚2.5133952498435974 /loss/FRt4/assignment_loss‚3.022376675605774 2loss/SRqqt2/assignment_loss‚1.9290745770558715 .loss/FRt1/detection_loss‚1.835486273516694 1loss/SRqqt2/detection_loss‚2.4316116025050483 _runtime‚9713.615833289 0loss/FBt3/assignment_loss‚0.5622814553789794 %loss/total_loss‚42.96252355276043 0loss/FRt3/assignment_loss‚1.3855980682373046 /loss/FRt3/detection_loss‚1.9063084012269973 1loss/SRqqt3/detection_loss‚1.1448555590246212ä0‡^ô#Ø 2loss/SRqqt3/assignment_loss‚1.1088233074808325 2loss/SRqqt4/assignment_loss‚1.9770086467266084 /loss/FRt4/detection_loss‚3.1640046644210815 0loss/SRqqt1/detection_loss‚1.678294860240486 _step‚677 0loss/FRt1/assignment_loss‚2.5133952498435974 /loss/FRt4/assignment_loss‚3.022376675605774 2loss/SRqqt2/assignment_loss‚1.9290745770558715 .loss/FRt1/detection_loss‚1.835486273516694 1loss/SRqqt2/detection_loss‚2.4316116025050483 _runtime‚9713.615833289 0loss/FBt3/assignment_loss‚0.5622814553789794 %loss/total_loss‚42.96252355276043 0loss/FRt3/assignment_loss‚1.3855980682373046 /loss/FRt3/detection_loss‚1.9063084012269973 1loss/SRqqt3/detection_loss‚1.1448555590246212 1loss/FBt2/assignment_loss‚0.16313559784170398 /loss/FBt1/detection_loss‚0.7537185139281832 .loss/FBt2/detection_loss‚2.729530772992543 2loss/SRqqt1/assignment_loss‚1.1853086085369189 /loss/FBt3/detection_loss‚3.0339554250240326 .loss/FBt4/detection_loss‚2.142945820635015 epoch‚6 trainer/global_step‚8249 % _timestamp‚1.7885056558388312e+09 /loss/FRt2/assignment_loss‚2.901768916032531 1loss/FBt1/assignment_loss‚0.12025907197396239 -loss/FRt2/detection_loss‚2.18857678711753 0loss/FBt4/assignment_loss‚0.5021955508032182 1loss/SRqqt4/detection_loss‚2.5820091466108956¥NžVútõ#‚(:é ¾ÜéÔŠŽÁm" disk.sdb2.out‚1082.69970703125 disk.nvme0n1.in‚ 939.5390625# disk.nvme0n1.out‚107492.6015625 disk.md0.in‚ 12.296875 disk.rbd2.in‚0) proc.memory.availableMB‚ 1317.12890625 network.sent‚ 2.1478763e+07 disk.sdb1.out‚0# disk.nvme1n1.out‚38976.60546875* proc.memory.percent‚18.338489532470703 disk.sda.in‚8.515625 disk.sda1.out‚0 disk.rbd0.out‚ 2.74609375 disk.sdb1.in‚0 disk.md0p1.in‚ 12.296875 disk.md0.out‚ 1041.46484375! proc.memory.rssMB‚ 3004.578125) disk./.usagePercent‚75.77469453703527 disk.rbd0.in‚0 disk.md0p1.out‚ 1041.46484375 disk.nvme1n1.in‚ 744.32421875 disk.sdb2.in‚3.78125 proc.cpu.threads‚17! disk.sda.out‚1082.69970703125 disk.sdb.in‚3.78125 disk.rbd2.out‚ 3.033203125 disk.sda2.in‚8.515625" disk.sda2.out‚1082.69970703125 disk.sda1.in‚0 disk.dm-0.in‚ 1683.86328125! disk.dm-0.out‚146469.20703125! disk.sdb.out‚1082.69970703125$ memory_percent‚91.96088314056396 cpu‚25.181139248018816 network.recv‚ 3.155934e+06$ disk./.usageGB‚2710.070110321045Iuf’ö#‚(:‡ ¾ÜéÔòý¤n gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚118.142' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.5355 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t+;‘÷#‚(:† ÍÜéÔ“Âm gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚180.29' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.0725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|’@ðeø#‚(:Ú ÍÜéÔ·Àém disk.md0.out‚ 1043.05859375! proc.memory.rssMB‚ 3004.578125 cpu‚25.180910408652686 disk.sda2.out‚1084.361328125 disk.md0.in‚ 12.296875 network.recv‚ 3.160358e+06) disk./.usagePercent‚75.77679660280232 disk.sdb2.in‚3.78125 disk.rbd0.out‚ 2.74609375 disk.sdb1.out‚0 disk.md0p1.out‚ 1043.05859375$ disk./.usageGB‚2710.145290374756 disk.sdb2.out‚1084.361328125 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.dm-0.in‚ 1683.86328125# disk.nvme1n1.out‚39053.79296875 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 1318.0859375 disk.sda2.in‚8.515625 disk.nvme1n1.in‚ 744.32421875 disk.sdb.in‚3.78125 disk.sdb.out‚1084.361328125$ memory_percent‚91.95504188537598 network.sent‚ 2.1574901e+07 disk.sda.out‚1084.361328125 disk.dm-0.out‚146646.5$ disk.nvme0n1.out‚107592.70703125 disk.rbd0.in‚0 disk.md0p1.in‚ 12.296875 disk.rbd2.out‚ 3.033203125* proc.memory.percent‚18.338489532470703 disk.sda.in‚8.515625 disk.nvme0n1.in‚ 939.5390625ÿ (’ù#‚(:‡ ÜÜéÔäòl gpu.0.gpu‚41.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.942' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4855 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ðù:™jú#‚(:ß ÜÜéÔûèÑm disk.nvme0n1.in‚ 939.5390625( disk./.usagePercent‚75.7777385219615% disk./.usageGB‚2710.1789779663086* proc.memory.percent‚18.338489532470703 proc.cpu.threads‚17 disk.sdb2.in‚3.78125# disk.nvme0n1.out‚107702.3515625 disk.nvme1n1.in‚ 744.32421875 disk.md0.out‚1044.625 disk.rbd2.out‚ 3.033203125" disk.sdb2.out‚1086.00634765625 disk.md0.in‚ 12.296875 disk.sda1.in‚0 disk.sdb1.in‚0 disk.md0p1.in‚ 12.296875 disk.md0p1.out‚1044.625! proc.memory.rssMB‚ 3004.578125" disk.sda2.out‚1086.00634765625 network.recv‚ 3.164841e+06 disk.sda.in‚8.515625! disk.sda.out‚1086.00634765625 disk.sda1.out‚0! disk.sdb.out‚1086.00634765625 disk.sda2.in‚8.515625! disk.dm-0.out‚146758.72265625) proc.memory.availableMB‚ 1317.70703125 disk.sdb1.out‚0 disk.rbd0.out‚ 2.74609375 disk.sdb.in‚3.78125 disk.rbd2.in‚0$ memory_percent‚91.95735454559326 network.sent‚ 2.1580876e+07 disk.rbd0.in‚0 disk.dm-0.in‚ 1683.86328125# disk.nvme1n1.out‚39056.37109375 cpu‚25.180684964210634#eßû#‚(:„ ëÜéÔðµ²m gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.56' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.14 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{?üuü#‚(:ê ëÜéÔˆÚšn) proc.memory.availableMB‚ 1317.60546875 disk.md0.out‚ 1046.52734375) disk./.usagePercent‚75.78036493090445 disk.sda1.in‚0 disk.md0p1.out‚ 1046.52734375 proc.cpu.threads‚17% disk./.usageGB‚2710.2729110717773 disk.md0p1.in‚ 12.296875* proc.memory.percent‚18.338489532470703" disk.sdb2.out‚1087.97119140625 disk.rbd0.in‚0 disk.dm-0.out‚147078.9609375 disk.sdb.in‚3.78125 disk.sda2.in‚8.515625 disk.nvme0n1.in‚ 939.5390625 disk.sdb2.in‚3.78125 disk.sda.in‚8.515625! disk.sda.out‚1087.97119140625" disk.sda2.out‚1087.97119140625$ disk.nvme0n1.out‚107964.57421875 network.sent‚ 2.1586892e+07 network.recv‚ 3.169106e+06 disk.sda1.out‚0 disk.sdb1.in‚0 disk.md0.in‚ 12.296875 disk.rbd2.out‚ 3.033203125 cpu‚25.180456141617217 disk.sdb1.out‚0 disk.rbd0.out‚ 2.74609375 disk.dm-0.in‚ 1683.86328125 disk.nvme1n1.in‚ 744.32421875 disk.rbd2.in‚0! proc.memory.rssMB‚ 3004.578125! disk.sdb.out‚1087.97119140625# disk.nvme1n1.out‚39114.38671875$ memory_percent‚91.95797443389893%ç©|2ý#- _wandbruntime‚9770  _runtime‚9770OxWÇþ#Á trainer/global_step‚8299 % _timestamp‚1.7885057122458677e+09 _runtime‚9770.023034939 _step‚678 & lr-AdamW/pg1‚0.0002849520614221541 & lr-AdamW/pg2‚0.0002849520614221541|“©Ìÿ#Æ & lr-AdamW/pg1‚0.0002849520614221541 & lr-AdamW/pg2‚0.0002849520614221541 trainer/global_step‚8299 % _timestamp‚1.7885057122458677e+09 _runtime‚9770.023034939 _step‚678¦BŒ-Ï2€$- _wandbruntime‚9770  _runtime‚9770rr y"$øO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚707468285218816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚707469213193216 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 210012879 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚780079864824320 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚646 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 210012879 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚684388840685718 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 173188720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9460811570146 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚780080125935104 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _runtime‚9770.024086454 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50195680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚679 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 209986535 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9496321202688 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 383201601 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50196326 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 % _timestamp‚1.7885057122465606e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚707469213193216 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚707468285218816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 209986535 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚697972891990528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237170688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 173188720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9496321202688 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚684389765700082 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 173188722 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9495411231232 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚770163496629760 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚697972873987584 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚697972873987584 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9916629305344 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 173188722 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚8299 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153956664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.largs˜…e_pool.freed‚ 103760984 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153957312 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚770163472689664 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚674928937090576 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚648 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103760986 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9459903595142 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9495411231232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0# ›c(‚$ýO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 173188722 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚8299 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153956664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 103760984 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153957312 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚770163472689664 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚674928937090576 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚648 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103760986 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9459903595142 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9495411231232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚707468285218816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚707469213193216 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 210012879 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚780079864824320 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚646 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 210012879 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚684388840685718 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 173188720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9460811570146 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚780080125935104 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _runtime‚9770.024086454 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50195680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚679 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 209986535 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9496321202688 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 383201601 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50196326 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 % _timestamp‚1.7885057122465606e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚707469213193216 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚707468285218816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 209986535 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚697972891990528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237170688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 173188720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9496321202688 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚684389765700082 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 173188722 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9495411231232 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚770163496629760 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚697972873987584 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚697972873987584 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9916629305344§æðÔ—2ƒ$- _wandbruntime‚9771  _runtime‚9771N¯]'„$€N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 383246618 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 173208882 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚698054493073408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9461928843632 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9496532691456 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153975190 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 trainer/global_step‚8299 _step‚680 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚698054493073408 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153974543 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚647 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚645 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚675007971724480 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 383220246 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 173208880 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 383220246 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 210037736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚698054511076352 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 173208882 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 210011366 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚770253698394112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50202139 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9917798316032 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚684469900568112 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚770253722334208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 383246618 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9461020868420 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚707551953752576 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚707551953752576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9497442676224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚675007954685120 $ _timestamp‚1.788505713368475e+09 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚780171520650240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚01Ö| '…$…N E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50202139 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9917798316032 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚684469900568112 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚770253722334208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 383246618 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9461020868420 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚707551953752576 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚707551953752576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9497442676224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚675007954685120 $ _timestamp‚1.788505713368475e+09 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚780171520650240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 383246618 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 173208882 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚698054493073408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9461928843632 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9496532691456 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153975190 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 trainer/global_step‚8299 _step‚680 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚698054493073408 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153974543 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚647 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚645 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚675007971724480 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 383220246 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 173208880 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 383220246 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 210037736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚698054511076352 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 173208882 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 210011366 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚770253698394112¨ziH2†$- _wandbruntime‚9771  _runtime‚9771§¤ªÒ‡$Ö 1loss/SRqqt1/detection_loss‚2.2011799175163795 1loss/SRqqt2/detection_loss‚2.5133977143660835 trainer/global_step‚8299 /loss/FRt1/assignment_loss‚2.249836725267497 2loss/SRqqt2/assignment_loss‚1.7202918957109037 1loss/FBt1/assignment_loss‚0.15654137562102313 _runtime‚9771.148809027 0loss/FBt2/assignment_loss‚0.2992350970666199 /loss/FBt1/detection_loss‚0.5811954719515947 /loss/FBt2/detection_loss‚3.7914284893444608 .loss/FBt4/detection_loss‚2.410848924091884 % _timestamp‚1.7885057133717194e+09 1loss/SRqqt4/assignment_loss‚1.845415285229683 _step‚681 0loss/FRt4/assignment_loss‚2.6678555632630983 .loss/FRt2/detection_loss‚2.554689944100877 /loss/FRt3/assignment_loss‚1.614196858622811 2loss/SRqqt1/assignment_loss‚1.4952294211192378 2loss/SRqqt3/assignment_loss‚1.0604464050830127 /loss/FRt3/detection_loss‚1.5903862117592134 epoch‚6 1loss/FBt3/assignment_loss‚0.08345883756`üöåŠ798382 .loss/FRt4/detection_loss‚3.290662370622158 1loss/SRqqt3/detection_loss‚0.7846474540921358 1loss/FBt4/assignment_loss‚0.26035259903174585 /loss/FRt1/detection_loss‚1.7801028217072599 0loss/FRt2/assignment_loss‚2.6572055160999297 0loss/SRqqt4/detection_loss‚2.683609507083893 /loss/FBt3/detection_loss‚2.6623530983924866 &loss/total_loss‚42.954567504711974Z°^aˆ$Û 1loss/FBt4/assignment_loss‚0.26035259903174585 /loss/FRt1/detection_loss‚1.7801028217072599 0loss/FRt2/assignment_loss‚2.6572055160999297 0loss/SRqqt4/detection_loss‚2.683609507083893 /loss/FBt3/detection_loss‚2.6623530983924866 &loss/total_loss‚42.954567504711974 1loss/SRqqt1/detection_loss‚2.2011799175163795 1loss/SRqqt2/detection_loss‚2.5133977143660835 trainer/global_step‚8299 /loss/FRt1/assignment_loss‚2.249836725267497 2loss/SRqqt2/assignment_loss‚1.7202918957109037 1loss/FBt1/assignment_loss‚0.15654137562102313 _runtime‚9771.148809027 0loss/FBt2/assignment_loss‚0.2992350970666199 /loss/FBt1/detection_loss‚0.5811954719515947 /loss/FBt2/detection_loss‚3.7914284893444608 .loss/FBt4/detection_loss‚2.410848924091884 % _timestamp‚1.7885057133717194e+09 1loss/SRqqt4/assignment_loss‚1.845415285229683 _step‚681 0loss/FRt4/assignment_loss‚2.6678555632630983 .loss/FRt2/detection_loss‚2.554689944100877 /loss/FRt3/assignment_loss‚1.614196858622811 2loss/SRqqt1/assignment_loss‚1.4952294211192378 2loss/SRqqt3/assignment_loss‚1.0604464050830127 /loss/FRt3/detection_loss‚1.5903862117592134 epoch‚6 1loss/FBt3/assignment_loss‚0.08345883756798382 .loss/FRt4/detection_loss‚3.290662370622158 1loss/SRqqt3/detection_loss‚0.7846474540921358©A:“‰$‚(:ˆ úÜéÔæôm gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚177.411' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.35275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\TmŠ$‚(:â úÜéÔ™Š²n disk.md0p1.out‚ 1047.8671875 disk.sdb.in‚3.78125 disk.sda2.in‚8.515625 disk.nvme0n1.in‚ 939.5390625 disk.md0.in‚ 12.296875 network.sent‚ 2.1603711e+07 disk.sda.in‚8.515625! disk.sda.out‚1089.38720703125 disk.rbd2.out‚ 3.033203125 cpu‚25.180230095625877$ disk./.usageGB‚2710.398349761963 disk.rbd0.out‚ 2.74609375 disk.dm-0.in‚ 1683.86328125" disk.sda2.out‚1089.38720703125" disk.nvme1n1.out‚ 39114.6484375 disk.rbd2.in‚0! proc.memory.rssMB‚ 3004.578125 network.recv‚ 3.173543e+06 disk.sdb2.in‚3.78125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.nvme1n1.in‚ 744.32421875! disk.nvme0n1.out‚ 107964.78125 disk.md0.out‚ 1047.8671875$ memory_percent‚91.95828437805176* proc.memory.percent‚18.338489532470703 disk.rbd0.in‚0! disk.sdb.out‚1089.38720703125 proc.cpu.threads‚17" disk.sdb2.out‚1089.38720703125 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.in‚ 12.296875 disk.dm-0.out‚147079.4296875( proc.memory.availableMB‚ 1317.5546875) disk./.usagePercent‚75.78387224918221NГ‹$‚(:ˆ ‰ÝéÔÅÅ·m gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚172.395' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.09875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mâŠmnŒ$‚(:ã ‰ÝéÔ¡¿¨n disk.sdb2.in‚3.78125 disk.sda.out‚1090.7685546875 disk.sda1.in‚0 disk.md0p1.in‚ 12.296875 disk.sda2.in‚8.515625! disk.sda2.out‚1090.7685546875 disk.nvme1n1.in‚ 744.32421875 disk.sdb.in‚3.78125' proc.memory.availableMB‚ 1317.546875 cpu‚25.179980661667905 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 1683.86328125$ disk.nvme0n1.out‚108227.19921875 proc.cpu.threads‚17 disk.nvme0n1.in‚ 939.5390625* proc.memory.percent‚18.338489532470703 disk.sda.in‚8.515625 disk.sda1.out‚0 disk.rbd0.out‚ 2.74609375$ memory_percent‚91.95833206176758# disk.nvme1n1.out‚39345.85546875) disk./.usagePercent‚75.79084102170893! disk.sdb2.out‚1090.7685546875 disk.md0p1.out‚ 1049.18359375 disk.dm-0.out‚147573.0546875 disk.sdb.out‚1090.7685546875 disk.md0.out‚ 1049.18359375! proc.memory.rssMB‚ 3004.578125 network.sent‚ 2.1700077e+07% disk./.usageGB‚2710.6475868225098 disk.rbd2.in‚0 disk.md0.in‚ 12.296875 network.recv‚ 3.17819e+06 disk.sdb1.in‚0 disk.rbd2.out‚ 3.033203125Ö“$‚(:ˆ ˜ÝéÔÒøl gpu.0.gpu‚54.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚178.117' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.52925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\örŽ$‚(:ç ˜ÝéÔÒ›úm disk.sda1.in‚0 disk.rbd0.in‚0 disk.sdb1.in‚0 cpu‚25.179758955685873! disk.sda.out‚1092.53759765625 disk.sda2.in‚8.515625" disk.sdb2.out‚1092.53759765625 disk.md0p1.out‚ 1050.88671875! disk.sdb.out‚1092.53759765625" disk.sda2.out‚1092.53759765625 disk.nvme1n1.in‚ 744.32421875! proc.memory.rssMB‚ 3004.578125 disk.sda1.out‚0 disk.sdb1.out‚0 disk.dm-0.out‚147865.5703125$ memory_percent‚91.95985794067383* proc.memory.percent‚18.338489532470703$ disk./.usageGB‚2710.647789001465 disk.sda.in‚8.515625 disk.md0.in‚ 12.296875 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 1683.86328125 disk.sdb.in‚3.78125 disk.md0.out‚ 1050.88671875 network.sent‚ 2.1706057e+07 disk.rbd2.out‚ 3.033203125 network.recv‚ 3.182433e+06) disk./.usagePercent‚75.79084667471712 disk.rbd0.out‚ 2.74609375 disk.md0p1.in‚ 12.296875 disk.nvme0n1.in‚ 939.5390625$ disk.nvme0n1.out‚108375.23046875# disk.nvme1n1.out‚39490.33984375' proc.memory.availableMB‚ 1317.296875 disk.sdb2.in‚3.78125¼¡üi$‚(:Þ §ÝéÔÀšm! disk.dm-0.out‚148112.74609375 disk.nvme0n1.in‚ 939.5390625 cpu‚25.17953997492034 proc.cpu.threads‚17 disk.md0.out‚1052.875 disk.rbd2.out‚ 3.033203125 disk.rbd0.in‚0" disk.sda2.out‚1094.60107421875) proc.memory.availableMB‚ 1317.67578125" disk.nvme0n1.out‚ 108541.890625$ memory_percent‚91.95754528045654* proc.memory.percent‚18.338489532470703 disk.md0.in‚ 12.296875 disk.rbd2.in‚0" disk.sdb2.out‚1094.60107421875 disk.sda.in‚8.515625 disk.sda1.in‚0 disk.rbd0.out‚ 2.74609375 disk.sdb.in‚3.78125! disk.sdb.out‚1094.60107421875 network.recv‚ 3.185905e+06 disk.nvme1n1.in‚ 744.32421875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 1683.86328125! proc.memory.rssMB‚ 3004.578125 network.sent‚ 2.1711087e+07# disk.nvme1n1.out‚39570.85546875 disk.sdb2.in‚3.78125 disk.sda1.out‚0) disk./.usagePercent‚75.79084774132244 disk.sda2.in‚8.515625% disk./.usageGB‚2710.6478271484375! disk.sda.out‚1094.60107421875 disk.md0p1.in‚ 12.296875 disk.md0p1.out‚1052.875Žã»’$‚(:‡ §ÝéÔâÈÔ{ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.598' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.3995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`x€2‘$- _wandbruntime‚9827  _runtime‚9827ZO /É’$à ' lr-AdamW/pg1‚0.00027822673863108415 ' lr-AdamW/pg2‚0.00027822673863108415 trainer/global_step‚8349 % _timestamp‚1.7885057700347426e+09 _runtime‚9827.811955224 _step‚682û_AΓ$È ' lr-AdamW/pg1‚0.00027822673863108415 ' lr-AdamW/pg2‚0.00027822673863108415 trainer/global_step‚8349 % _timestamp‚1.7885057700347426e+09 _runtime‚9827.811955224 _step‚682ªùÓ_2”$- _wandbruntime‚9827  _runtime‚9827`Yeþ'•$øO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 385428157 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 211257781 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚784650060452864 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚784649818216448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9551498656256 _step‚683 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚702053846281728 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50486326 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 211231437 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9975278600704 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚774674781852160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚688395597668026 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50486970 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚678879833857136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚711606254909440 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 % _timestamp‚1.7885057700358217e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚774674757912064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚711605326935040 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 104364234 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9552408627712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218296320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚9827.813109089 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 174196720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 211257781 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚711605326935040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9516688825254 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 154851206 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 385454503 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 174196720 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 211231437 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242236416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 104364236 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9515780850250 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 trainer/global_step‚8349 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 174196722 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚702053828278784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570W&Áø(–$ýO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9552408627712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218296320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚9827.813109089 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 174196720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 211257781 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚711605326935040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9516688825254 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 154851206 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 385454503 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 174196720 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 211231437 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242236416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 104364236 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9515780850250 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 trainer/global_step‚8349 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 174196722 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚702053828278784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 385428157 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 211257781 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚784650060452864 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚784649818216448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9551498656256 _step‚683 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚702053846281728 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50486326 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 211231437 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9975278600704 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚774674781852160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚688395597668026 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50486970 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚678879833857136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚711606254909440 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 % _timestamp‚1.7885057700358217e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚774674757912064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚711605326935040 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 104364234 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 174216880 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚702135465367552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚784741462800896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 385473148 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚711688995468800 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9517806098740 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 211256268 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚711688067481088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚784741216383488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚774765007556608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9552620116480 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚688476657550420 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 _runtime‚9828.920266623 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚678958851451680 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 174216882 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _step‚684 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 174216880 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚678958834412320 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚688475732535848 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚702135447364608 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚702135447364608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 104376299 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9976232766976 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9553530101248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚8349 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚702135465367552 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9976455244288 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚774764983616512 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 % _timestamp‚1.7885057711422126e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 385499520 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 385499520 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 154869031 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154868398 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185¶ ^ '™$‡N TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 211282638 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚711688995468800 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 174216882 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _step‚684 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 174216880 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚678958834412320 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚688475732535848 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚702135447364608 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚702135447364608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 104376299 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9976232766976 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9553530101248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚8349 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚702135465367552 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9976455244288 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚774764983616512 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 % _timestamp‚1.7885057711422126e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 385499520 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 385499520 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 154869031 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154868398 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 174216880 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚702135465367552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚784741462800896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 385473148 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚711688995468800 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9517806098740 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 211256268 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚711688067481088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚784741216383488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚774765007556608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9552620116480 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚688476657550420 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 _runtime‚9828.920266623 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚678958851451680 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚715742368651264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚8399 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚706134800572928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9607586081280 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 212476339 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9608496052736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216199168 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚789220191943168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚715743296625664 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 175204720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10033908669440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚706134800572928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9607586081280 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 155744326 YEN× ïDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚692403279664698 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 175204722 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50776214 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 387681059 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 175204722 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚706134782569984 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚682830713584336 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚682830696544976 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570ߌËë(¨$ÿO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10033908669440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚706134800572928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9607586081280 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 155744326 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚692403279664698 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 175204722 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50776214 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 387681059 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 175204722 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚706134782569984 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚682830713584336 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚682830696544976 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 % _timestamp‚1.7885058252622144e+09 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9572566080362 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚715742368651264 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚706134782569984 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 212502683 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50776840 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚715742368651264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚8399 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚706134800572928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9607586081280 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 212476339 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9608496052736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216199168 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚789220191943168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚715743296625664 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 175204720¯ w!2©$- _wandbruntime‚9883  _runtime‚9883L9}’ 'ª$ƒN UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9608707541504 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8399 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 $ _timestamp‚1.788505826200189e+09 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 104979549 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚715825109197312 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚789311364184064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10035309433856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚9883.977986194 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 175224882 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9609617526272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 387726050 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 155762042 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155761418 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚682909731178880 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50782491 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚715826037185024 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚715826037185024 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚706216419658752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 387726050 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 212501170 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚706216401655808 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚706216419658752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9608707541504 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚622 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50781869 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚789311602212864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 175224882 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 104979551 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚706216401655808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 387752422 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 212527540 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚692483414532728 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9609617526272 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚682909714139520 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 387752422 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 212527540 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚692483414532728 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9609617526272 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚682909714139520 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8399 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 $ _timestamp‚1.788505826200189e+09 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 104979549 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚715825109197312 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚789311364184064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10035309433856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚9883.977986194 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 175224882 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9609617526272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 387726050 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 155762042 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155761418 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚682909731178880 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50782491 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚715826037185024 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚715826037185024 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚706216419658752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 387726050 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 212501170 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚706216401655808 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚706216419658752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9608707541504 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚622 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50781869 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚789311602212864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 175224882 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 104979551 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0°„9Üþ2¬$- _wandbruntime‚9883  _runtime‚9883”K`­$Ï 0loss/FRt4/assignment_loss‚3.6111262987057366 1loss/SRqqt3/detection_loss‚0.8391161995729611 &loss/total_loss‚44.633843048126735 /loss/FRt3/detection_loss‚1.5676192147418475 /loss/FRt1/assignment_loss‚2.273059755563736 2loss/SRqqt1/assignment_loss‚1.2896512399117153 .loss/FBt1/assignment_loss‚0.15264785172021 /loss/FBt2/detection_loss‚1.7499884143471718 % _timestamp‚1.7885058262024724e+09 0loss/FRt3/assignment_loss‚1.4119787284961114 1loss/FBt3/assignment_loss‚0.46685223855699104 0loss/FBt4/assignment_loss‚0.1773600827212001 1loss/SRqqt1/detectioߟ”õn_loss‚2.1115070591370264 .loss/FBt1/detection_loss‚0.675801496544423 .loss/FBt3/detection_loss‚4.706238150596619 1loss/FBt2/assignment_loss‚0.10658410983467093 .loss/FRt1/detection_loss‚2.152113125970546 .loss/FRt4/detection_loss‚3.173199087381363 0loss/SRqqt2/detection_loss‚3.003135621547699 1loss/SRqqt4/detection_loss‚2.2751454428220406 /loss/FRt2/assignment_loss‚3.355735504313519 2loss/SRqqt4/assignment_loss‚1.6237024145129209 /loss/FRt2/detection_loss‚2.3686544179034077 .loss/FBt4/detection_loss‚2.341616630554199 trainer/global_step‚8399 epoch‚6 1loss/SRqqt2/assignment_loss‚2.345252042843236 1loss/SRqqt3/assignment_loss‚0.855757919827386 _runtime‚ 9883.97945195 _step‚689"ßwZ®$Ô .loss/FBt1/detection_loss‚0.675801496544423 .loss/FBt3/detection_loss‚4.706238150596619 1loss/FBt2/assignment_loss‚0.10658410983467093 .loss/FRt1/detection_loss‚2.152113125970546 .loss/FRt4/detection_loss‚3.173199087381363 0loss/SRqqt2/detection_loss‚3.003135621547699 1loss/SRqqt4/detection_loss‚2.2751454428220406 /loss/FRt2/assignment_loss‚3.355735504313519 2loss/SRqqt4/assignment_loss‚1.6237024145129209 /loss/FRt2/detection_loss‚2.3686544179034077 .loss/FBt4/detection_loss‚2.341616630554199 trainer/global_step‚8399 epoch‚6 1loss/SRqqt2/assignment_loss‚2.345252042843236 1loss/SRqqt3/assignment_loss‚0.855757919827386 _runtime‚ 9883.97945195 _step‚689 0loss/FRt4/assignment_loss‚3.6111262987057366 1loss/SRqqt3/detection_loss‚0.8391161995729611 &loss/total_loss‚44.633843048126735 /loss/FRt3/detection_loss‚1.5676192147418475 /loss/FRt1/assignment_loss‚2.273059755563736 2loss/SRqqt1/assignment_loss‚1.2896512399117153 .loss/FBt1/assignment_loss‚0.15264785172021 /loss/FBt2/detection_loss‚1.7499884143471718 % _timestamp‚1.7885058262024724e+09 0loss/FRt3/assignment_loss‚1.4119787284961114 1loss/FBt3/assignment_loss‚0.46685223855699104 0loss/FBt4/assignment_loss‚0.1773600827212001 1loss/SRqqt1/detection_loss‚2.1115070591370264±Ž>­<¯$‚(:’ ãÝéÔ篱n gpu.0.gpu‚15.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…*ª‚i°$‚(:Þ ãÝéÔ¨íðn network.sent‚ 2.1846765e+07 disk.md0p1.in‚ 12.296875! proc.memory.rssMB‚ 3004.578125 disk.rbd0.out‚ 2.74609375 disk.sdb.out‚1101.466796875 disk.nvme0n1.in‚ 939.6640625 disk.md0.in‚ 12.296875 disk.sdb2.in‚3.78125 disk.sdb1.in‚0 disk.sda2.out‚1101.466796875$ disk.nvme0n1.out‚109118.76953125 disk.md0.out‚ 1059.46484375 disk.sdb2.out‚1101.466796875 disk.sda.out‚1101.466796875 disk.sda1.out‚0 proc.cpu.threads‚17 network.recv‚ 3.2038e+06 disk.rbd2.out‚ 3.033203125 disk.sda.in‚8.515625 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 1684.23828125) disk./.usagePercent‚75.75851658750464% disk./.usageGB‚2709.4915084838867 disk.dm-0.out‚ 148852.09375 disk.nvme1n1.in‚ 744.57421875$ memory_percent‚91.96102619171143 cpu‚25.178620646357967 disk.sdb.in‚3.78125 disk.sda2.in‚8.515625# disk.nvme1n1.out‚39733.32421875 disk.rbd2.in‚0 disk.md0p1.out‚ 1059.46484375) proc.memory.availableMB‚ 1317.10546875* proc.memory.percent‚18.338489532470703a––C‘±$‚(:† òÝéÔ¯—‹m gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.204' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.551 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@÷T§q²$‚(:æ òÝéÔŒÀÂm% disk./.usageGB‚2709.7695808410645 disk.md0p1.in‚ 12.296875! disk.sdb.out‚1102.66943359375 network.recv‚ 3.209755e+06 disk.sda.in‚8.515625! disk.sda.out‚1102.66943359375 disk.sda1.out‚0 disk.sda2.in‚8.515625 disk.md0.out‚ 1060.6015625 disk.rbd2.in‚0! proc.memory.rssMB‚ 3004.578125 disk.sdb2.in‚3.78125" disk.sdb2.out‚1102.66943359375 disk.rbd0.in‚0 disk.rbd0.out‚ 2.74609375 network.sent‚ 2.1944388e+07 disk.sdb1.in‚0 disk.nvme0n1.in‚ 939.6640625 cpu‚25.17840830423979$ disk.nvme0n1.out‚109118.77734375 disk.nvme1n1.in‚ 744.57421875 disk.sda1.in‚0 disk.sdb.in‚3.78125 disk.rbd2.out‚ 3.033203125( proc.memory.availableMB‚ 1317.4453125 disk.sdb1.out‚0" disk.nvme1n1.out‚ 39843.0234375$ memory_percent‚91.95895195007324 proc.cpu.threads‚17) disk./.usagePercent‚75.76629160699365 disk.md0p1.out‚ 1060.6015625 disk.dm-0.in‚ 1684.23828125! disk.dm-0.out‚148961.80078125" disk.sda2.out‚1102.66943359375 disk.md0.in‚ 12.296875* proc.memory.percent‚18.338489532470703"¿;’³$‚(:‡ ÞéÔŽÝÀm gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚158.586' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.6465 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t-¾‚q´$‚(:æ ÞéÔ±âËn disk.sdb.in‚3.78125! disk.sdb.out‚1104.58544921875! proc.memory.rssMB‚ 3004.578125 disk.sdb1.out‚0 disk.md0p1.in‚ 12.296875 disk.dm-0.in‚ 1684.23828125 network.sent‚ 2.1949654e+07" disk.sdb2.out‚1104.58544921875 disk.md0p1.out‚ 1062.4609375 disk.rbd2.out‚ 3.033203125) disk./.usagePercent‚75.76976682045125 disk.sda1.out‚0 disk.sdb1.in‚0! disk.dm-0.out‚149213.74609375 disk.nvme0n1.in‚ 939.6640625# disk.nvme1n1.out‚40094.70703125 proc.cpu.threads‚17 disk.nvme1n1.in‚ 744.57421875 network.recv‚ 3.213202e+06 disk.sda.in‚8.515625! disk.sda.out‚1104.58544921875 disk.rbd0.out‚ 2.74609375" disk.sda2.out‚1104.58544921875 disk.rbd2.in‚0( proc.memory.availableMB‚ 1317.8203125$ disk./.usageGB‚2709.893871307373# disk.nvme0n1.out‚109119.0390625 disk.md0.out‚ 1062.4609375* proc.memory.percent‚18.338489532470703 disk.sda2.in‚8.515625 cpu‚25.178158566122608 disk.sdb2.in‚3.78125 disk.sda1.in‚0$ memory_percent‚91.95666313171387 disk.md0.in‚ 12.296875 disk.rbd0.in‚0w<®’µ$‚(:‡ ÞéÔÆ”ðl gpu.0.gpu‚57.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚161.254' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.3135 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 mm¶$‚(:â ÞéÔõõm* proc.memory.percent‚18.338489532470703! disk.sda.out‚1106.05419921875 disk.sda1.in‚0 disk.nvme0n1.in‚ 939.6640625 disk.md0.out‚ 1063.8828125 network.sent‚ 2.1955706e+07 disk.rbd2.out‚ 3.033203125 disk.rbd0.out‚ 2.74609375# disk.nvme0n1.out‚109119.0390625" disk.nvme1n1.out‚ 40186.2421875 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sdb2.in‚3.78125 disk.sdb1.out‚0 disk.dm-0.in‚ 1684.23828125 disk.md0.in‚ 12.296875% disk./.usageGB‚2710.1485481262207 disk.rbd0.in‚0 disk.md0p1.out‚ 1063.8828125! disk.sdb.out‚1106.05419921875 cpu‚25.17794390886092 proc.cpu.threads‚17 network.recv‚ 3.217412e+06" disk.sdb2.out‚1106.05419921875 disk.sda.in‚8.515625 disk.sdb.in‚3.78125$ memory_percent‚91.95752143859863( proc.memory.availableMB‚ 1317.6796875! proc.memory.rssMB‚ 3004.578125) disk./.usagePercent‚75.77688769089676 disk.sdb1.in‚0 disk.sda2.in‚8.515625" disk.sda2.out‚1106.05419921875 disk.md0p1.in‚ 12.296875 disk.dm-0.out‚ 149305.28125 disk.nvme1n1.in‚ 744.57421875Õ¯é62·$- _wandbruntime‚9940  _runtime‚9940(Ãp”Ÿ$¿ _step‚690 % lr-AdamW/pg1‚0.000264932501133453 % lr-AdamW/pg2‚0.000264932501133453 trainer/global_step‚8449 % _timestamp‚1.7885058824405456e+09 _runtime‚9940.217796598Œå0»Ê¹$Ä % lr-AdamW/pg1‚0.000264932501133453 % lr-AdamW/pg2‚0.000264932501133453 trainer/global_step‚8449 % _timestamp‚1.7885058824405456e+09 _runtime‚9940.217796598 _step‚690²W[¥H2º$- _wandbruntime‚9940  _runtime‚9940GFçÄ(»$úO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚710215754864128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 176212720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 213747585 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _step‚691 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9664583477760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9663673506304 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 trainer/global_step‚8449 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚719879410367488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10092529776128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚651 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51067337 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10092750169600 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚686781576272176 % _timestamp‚1.7885058824415853e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393472 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 213747585 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 176212722 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51066688 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9628443335470 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚719879410367488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚710215736861184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9663673506304 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 µiÏ(¼$ÿO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 213721241 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚783697352296960 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 105570734 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 176212722 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9664583477760 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚710215736861184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚696409111632642 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚686781593311536 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 156638073 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 176212720 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚719880338341888 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚710215754864128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚696410036647006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚793790102466560 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 213721241 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚651 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51067337 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10092750169600 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚686781576272176 % _timestamp‚1.7885058824415853e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393472 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 213747585 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 176212722 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51066688 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9628443335470 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚719879410367488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚710215736861184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9663673506304 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚710215754864128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 176212720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 213747585 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _step‚691 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9664583477760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9663673506304 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 trainer/global_step‚8449 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚719879410367488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10092529776128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 176232880 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚627 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 389978952 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 213746072 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚710297373949952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9629560608956 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚719963078901248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 390005324 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 389978952 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚696489246500464 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9628652633744 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 176232882 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pûM: ool.allocated‚ 213772442 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10093719864832 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9664794966528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9665704951296 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 176232880 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 105582801 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚793881516149248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156655237 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283008 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚686860610906080 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51072438 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚625 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 _runtime‚9941.245769055 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 213746072 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51073063 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚719963078901248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚783787554061312 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10093938147840 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185ƲCÜ'¿$‰N K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 213746072 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚710297373949952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9629560608956 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚719963078901248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 390005324 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 389978952 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚696489246500464 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9628652633744 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 176232882 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 213772442 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10093719864832 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9664794966528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9665704951296 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 176232880 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 105582801 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚793881516149248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156655237 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283008 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚686860610906080 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51072438 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚625 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 _runtime‚9941.245769055 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 213746072 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51073063 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚719963078901248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚783787554061312 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10093938147840 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 390005324 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 213772442 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 176232880 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚627 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 389978952´|>C2À$- _wandbruntime‚9941  _runtime‚9941by”Ý\Á$Ö 0loss/FBt2/assignment_loss‚0.4094251166655643 /loss/FRt2/detection_loss‚2.4437684422162804 0loss/SRqqt2/detection_loss‚2.971360948524977 % _timestamp‚1.7885058834697163e+09 _step‚693 /loss/FRt1/assignment_loss‚2.229372440413995 2loss/SRqqt4/assignment_loss‚1.5713265711634323 0loss/FBt4/assignment_loss‚0.3548804935585808 /loss/FRt3/detection_loss‚1.6982757807924198 1loss/SRqqt3/detection_loss‚0.7887348867079709 /loss/FRt4/detection_loss‚3.3209234714508056 1loss/SRqqt4/detection_loss‚2.3482337203519097 /loss/FBt3/detection_loss‚2.5991698916141805 0loss/FRt2/assignment_loss‚3.4181539886876156 2loss/SRqqt1/assignment_loss‚1.2890377568701903 2loss/SRqqt3/assignment_loss‚1.0603355651146558 1loss/FBt3/assignment_loss‚0.14514009749445206 .loss/FRt1/detection_loss‚2.026044673898303 /loss/FBt1/detection_loss‚0.7233331854581649 /loss/FBt2/detection_loss‚2.4046038389205933 .loss/FBt4/detection_loss‚2.455586601387371 /loss/FRt4/assignment_loss‚3.116975024342537 &loss/total_loss‚42.068204952923516 epoch‚6 trainer/global_step‚8449 _runtime‚ 9941.24676187 0loss/FRt3/assignment_loss‚1.4817052208460295 2loss/SRqqt2/assignment_loss‚1.2348159943756305 1loss/FBt1/assignment_loss‚0.09080469432119971 1loss/SRqqt1/detection_loss‚1.8861965477466582LH’ŸaÂ$Û /loss/FRt3/detection_loss‚1.6982757807924198 1loss/SRqqt3/detection_loss‚0.7887348867079709 /loss/FRt4/detection_loss‚3.3209234714508056 1loss/SRqqt4/detection_loss‚2.3482337203519097 /loss/FBt3/detection_loss‚2.5991698916141805 0loss/FRt2/assignment_loss‚3.4181539886876156 2loss/SRqqt1/assignment_loss‚1.2890377568701903 2loss/SRqqt3/assignment_loss‚1.0603355651146558 1loss/FBt3/assignment_loss‚0.14514009749445206 .loss/FRt1/detection_loss‚2.026044673898303 /loss/FBt1/detection_loss‚0.7233331854581649 /loss/FBt2/detection_loss‚2.4046038389205933 .loss/FBt4/detection_loss‚2.455586601387371 /loss/FRt4/assignment_loss‚3.116975024342537 &loss/total_loss‚42.068204952923516 epoch‚6 trainer/global_step‚8449 _runtime‚ 9941.24676187 0loss/FRt3/assignment_loss‚1.4817052208460295 2loss/SRqqt2/assignment_loss‚1.2348159943756305 1loss/FBt1/assignment_loss‚0.09080469432119971 1loss/SRqqt1/detection_loss‚1.8861965477466582 0loss/FBt2/assignment_loss‚0.4094251166655643 /loss/FRt2/detection_loss‚2.4437684422162804 0loss/SRqqt2/detection_loss‚2.971360948524977 % _timestamp‚1.7885058834697163e+09 _step‚693 /loss/FRt1/assignment_loss‚2.229372440413995 2loss/SRqqt4/assignment_loss‚1.5713265711634323 0loss/FBt4/assignment_loss‚0.3548804935585808µOAË“Ã$‚(:ˆ ŸÞéÔ”Åßl gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚164.989' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.24725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó6¨_nÄ$‚(:ã ŸÞéÔË”¥n disk.md0.in‚ 12.296875 cpu‚25.177699362858263 proc.cpu.threads‚17 disk.sda1.in‚0 disk.nvme0n1.in‚ 939.6640625! proc.memory.rssMB‚ 3004.578125" disk.sdb2.out‚1107.77880859375 disk.sdb1.out‚0" disk.sda2.out‚1107.77880859375# disk.nvme1n1.out‚40186.54296875* proc.memory.percent‚18.338489532470703 network.recv‚ 3.221598e+06 disk.md0p1.in‚ 12.296875 disk.md0.out‚ 1065.53125! disk.sda.out‚1107.77880859375 disk.sdb1.in‚0) proc.memory.availableMB‚ 1317.27734375 disk.sda.in‚8.515625 disk.dm-0.in‚ 1684.23828125 network.sent‚ 2.1972257e+07 disk.md0p1.out‚ 1065.53125$ disk.nvme0n1.out‚109119.24609375 disk.rbd2.in‚0 disk.rbd2.out‚ 3.033203125$ memory_percent‚91.95997714996338) disk./.usagePercent‚75.77864257663175 disk.sdb2.in‚3.78125 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 2.74609375 disk.sdb.in‚3.78125 disk.sda2.in‚8.515625 disk.nvme1n1.in‚ 744.57421875$ disk./.usageGB‚2710.211311340332 disk.dm-0.out‚149305.7890625! disk.sdb.out‚1107.77880859375¡ ‚8‘Å$‚(:† ®ÞéÔØÈ¡m gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.798' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.4495 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í*$nÆ$‚(:ã ®ÞéÔßþ°n network.recv‚ 3.22782e+06 disk.sda2.in‚8.515625 disk.rbd2.out‚ 3.033203125! disk.sda.out‚1109.01611328125$ memory_percent‚91.95921421051025 disk.nvme1n1.in‚ 744.57421875 disk.sda.in‚8.515625 disk.dm-0.out‚149507.6328125 cpu‚25.17858358266148# disk.nvme1n1.out‚40388.38671875 disk.rbd0.in‚0 disk.md0p1.out‚ 1066.703125! disk.sdb.out‚1109.01611328125) proc.memory.availableMB‚ 1317.40234375 proc.cpu.threads‚17" disk.sda2.out‚1109.01611328125 disk.nvme0n1.in‚ 939.6640625 disk.dm-0.in‚ 1684.23828125! proc.memory.rssMB‚ 3004.578125 disk.md0.in‚ 12.296875 disk.sdb1.in‚0 disk.sdb.in‚3.78125) disk./.usagePercent‚75.78038850288206$ disk./.usageGB‚2710.273754119873$ disk.nvme0n1.out‚109119.24609375 disk.md0.out‚ 1066.703125 disk.sdb1.out‚0* proc.memory.percent‚18.338489532470703 network.sent‚ 2.2069542e+07 disk.sdb2.in‚3.78125" disk.sdb2.out‚1109.01611328125 disk.sda1.in‚0 disk.rbd0.out‚ 2.74609375 disk.md0p1.in‚ 12.296875 disk.rbd2.in‚0 disk.sda1.out‚0U~ÍÍ“Ç$‚(:ˆ ½ÞéÔ™ÃÄm gpu.0.gpu‚49.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.333' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.33325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04:]È$‚(:Ò ½ÞéÔõ‹™o disk.sdb2.in‚3.78125 disk.sda1.in‚0 disk.sdb1.out‚0 disk.md0p1.out‚1068.625 disk.sdb.out‚1111.009765625 disk.nvme0n1.in‚ 939.6640625 disk.rbd2.out‚ 3.033203125 disk.md0.in‚ 12.296875 disk.rbd0.out‚ 2.74609375 disk.sda.in‚8.515625 disk.sda1.out‚0 disk.sdb.in‚3.78125' proc.memory.availableMB‚ 1307.796875 disk.dm-0.out‚149726.7734375 disk.rbd2.in‚0 proc.memory.rssMB‚3013.625 network.sent‚ 2.2072778e+07) disk./.usagePercent‚75.78387608896136 disk.sda.out‚1111.009765625 disk.sda2.out‚1111.009765625% disk./.usageGB‚2710.3984870910645* proc.memory.percent‚18.393707275390625 disk.sdb2.out‚1111.009765625 disk.rbd0.in‚0$ disk.nvme0n1.out‚109119.39453125# disk.nvme1n1.out‚40607.37890625$ memory_percent‚92.01784133911133 proc.cpu.threads‚17 disk.md0p1.in‚ 12.296875 disk.sda2.in‚8.515625 disk.nvme1n1.in‚ 744.57421875 disk.md0.out‚1068.625 cpu‚25.18446134771116 disk.sdb1.in‚0 network.recv‚ 3.231208e+06 disk.dm-0.in‚ 1684.238281256HÌHdÉ$‚(:Ù ÌÞéÔ²Þål network.sent‚ 2.2080915e+07 network.recv‚ 3.23577e+06 disk.sdb.out‚1112.400390625) proc.memory.availableMB‚ 1308.16015625 proc.cpu.threads‚17 disk.sdb.in‚3.78125 disk.md0.in‚ 12.296875 disk.md0p1.out‚ 1069.953125 disk.dm-0.in‚ 1684.23828125 disk.sda2.out‚1112.400390625 disk.sda.out‚1112.400390625 proc.memory.rssMB‚3013.625 disk.sda2.in‚8.515625 disk.md0.out‚ 1069.953125 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0$ disk./.usageGB‚2710.120922088623 disk.nvme0n1.in‚ 939.6640625 disk.sda.in‚8.515625 disk.rbd0.out‚ 2.74609375 disk.md0p1.in‚ 12.296875 disk.nvme1n1.in‚ 744.57421875 disk.sdb2.out‚1112.400390625! disk.dm-0.out‚149930.51171875* proc.memory.percent‚18.393707275390625) disk./.usagePercent‚75.77611525532313 disk.rbd2.out‚ 3.033203125 disk.sdb2.in‚3.78125 disk.sdb1.in‚0 disk.sdb1.out‚0$ memory_percent‚92.01562404632568$ disk.nvme0n1.out‚109119.39453125" disk.nvme1n1.out‚ 40811.1171875 cpu‚25.190413053248726}ÅÔ’Ê$‚(:‡ ÌÞéÔ̇m gpu.0.gpu‚61.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.334' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.8335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷³jÛ’Ë$‚(:‡ ÛÞéÔùåk gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.297' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.57425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zwA~oÌ$‚(:ä ÛÞéÔÛ§ôm) disk./.usagePercent‚75.77472226877362 disk.dm-0.out‚149931.2421875" disk.sda2.out‚1113.82275390625 disk.md0.in‚ 12.296875* proc.memory.percent‚18.393707275390625 disk.rbd0.out‚ 2.74609375 disk.sda2.in‚8.515625 disk.nvme0n1.in‚ 939.6640625! disk.sda.out‚1113.82275390625$ memory_percent‚92.01865196228027 network.recv‚ 3.239934e+06 disk.sdb1.out‚0! disk.sdb.out‚1113.82275390625 disk.md0.out‚ 1071.30859375 proc.memory.rssMB‚3013.625 disk.sda1.in‚0 disk.md0p1.in‚ 12.296875 disk.rbd2.out‚ 3.033203125$ disk./.usageGB‚2710.071102142334 disk.md0p1.out‚ 1071.30859375 disk.sdb.in‚3.78125 disk.rbd2.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 1684.23828125 disk.nvme1n1.in‚ 744.57421875 disk.sdb2.in‚3.78125" disk.sdb2.out‚1113.82275390625 disk.sda.in‚8.515625 proc.cpu.threads‚17 disk.rbd0.in‚0( proc.memory.availableMB‚ 1307.6640625 cpu‚25.19672749583386 network.sent‚ 2.2084617e+07 disk.sdb1.in‚0$ disk.nvme0n1.out‚109119.39453125# disk.nvme1n1.out‚40811.84765625…쎑Í$‚(:† êÞéÔãé˜m gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.3665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€š,JpÎ$‚(:å êÞéÔÂþ½m proc.memory.rssMB‚3013.625* proc.memory.percent‚18.393707275390625 disk.nvme1n1.in‚ 744.57421875 disk.rbd2.out‚ 3.033203125 disk.dm-0.in‚ 1684.23828125 disk.md0.out‚ 1073.38671875 disk.sda.in‚8.515625! disk.sda.out‚1115.97802734375 disk.sda1.in‚0 disk.md0p1.out‚ 1073.38671875 cpu‚25.20295648322939 network.sent‚ 2.2090934e+07 disk.sdb1.in‚0 disk.md0p1.in‚ 12.296875 disk.sdb.in‚3.78125 proc.cpu.threads‚17 disk.rbd2.in‚0) proc.memory.availableMB‚ 1308.16796875% disk./.usageGB‚2710.6482124328613! disk.nvme1n1.out‚ 40984.078125" disk.sdb2.out‚1115.97802734375 network.recv‚ 3.244753e+06) disk./.usagePercent‚75.79085851403619 disk.sda1.out‚0 disk.rbd0.in‚0$ memory_percent‚92.01557636260986 disk.sdb2.in‚3.78125 disk.rbd0.out‚ 2.74609375! disk.dm-0.out‚150180.55078125! disk.sdb.out‚1115.97802734375 disk.nvme0n1.in‚ 939.6640625 disk.md0.in‚ 12.296875 disk.sdb1.out‚0 disk.sda2.in‚8.515625" disk.sda2.out‚1115.97802734375$ disk.nvme0n1.out‚109196.47265625"¹2¸4Ï$/ _wandbruntime‚10020  _runtime‚10020¡“mâVÐ$âT ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 681067448 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13205766144 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1185 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 9703619335680 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 214505929 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 14135738368 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚714017206509056 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚723721508883968 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 190247424 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 683039232 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚700026743381506 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚723720807841792 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 391759213 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚714017188506112 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 9704302374912 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14401142784 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚570 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 13002869760 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189768192 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚51420696 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚690358701749808 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚723721508883968 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚714017206509056 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 106208184 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 166307328 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 177253286 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 391779926 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 214526640 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14401142784 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚570 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 698106808 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 9667343524890 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 157628657 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚714017188506112 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 214526640 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚787717598354944 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13684416521 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚615 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚221 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 409073152 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 trainer/global_step‚8462 _step‚694 ZLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚ 106208182 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 9703619335680 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 157628880 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚223 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚10133286291456 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚700026045274698 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1187728393 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32006 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚44 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 214505929 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚10133452598784 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 701042176 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚51420475 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 701042176 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚797850884646400 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚787717622295040 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 701042176 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚51420475 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 701042176 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚797850884646400 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚787717622295040 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚723721508883968 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚714017206509056 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 106208184¶Q¹~ö4Ò$/ _wandbruntime‚10021  _runtime‚10021ñ‚” r)Ó$ìR ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1185 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚225 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 9704109186560 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 683039232 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚615 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 190247424 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚714071467416064 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 9704792225792 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚44 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 trainer/global_step‚8462 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 177268762 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚714071467416064 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚223 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚787773865908736 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13684416521 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚723775558599680 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 9667831735774 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1185 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135738368 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 177268760 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 701042176 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚723776259641856 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚714071449413120 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 9704792225792 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 157644006 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 177268762 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14401142784 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 _runtime‚10021.377084519 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚714071449413120 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 9704109186560 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 166307328 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 214535454 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698106808 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚700076959590766 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚615 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 214514743 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚723776259641856 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚797907832298496 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 177268760 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚797907642051072 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 % _timestamp‚1.7885059635989943e+09 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚10133942449664 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14401142784M<¼¸Ô$ñR `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚714071467416064 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 9704792225792 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚44 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 trainer/global_step‚8462 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 177268762 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚714071467416064 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚223 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚787773865908736 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13684416521 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚723775558599680 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 9667831735774 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1185 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135738368 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 177268760 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 701042176 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚723776259641856 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚714071449413120 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 9704792225792 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 157644006 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 177268762 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14401142784 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 _runtime‚10021.377084519 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚714071449413120 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 9704109186560 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 166307328 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 214535454 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698106808 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚700076959590766 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚615 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 214514743 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚723776259641856 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚797907832298496 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 177268760 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚797907642051072 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 % _timestamp‚1.7885059635989943e+09 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚10133942449664 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14401142784 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1185 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚225 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 9704109186560 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 683039232 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚615 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 190247424 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 14135738368·G@YÑ’Õ$‚(:‡ ùÞéÔÈ¥m gpu.0.gpu‚18.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.798' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.4495 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`vNükÖ$‚(:à ùÞéÔÊ×¢n) proc.memory.availableMB‚ 1309.04296875 cpu‚25.209384018686148! disk.sda.out‚1117.42724609375 disk.rbd0.out‚ 2.74609375 disk.md0p1.out‚ 1074.78125 proc.cpu.threads‚17 disk.nvme1n1.in‚ 744.57421875 proc.memory.rssMB‚3013.625 disk.sdb.in‚3.78125! disk.sdb.out‚1117.42724609375 disk.sda2.in‚8.515625 disk.md0.in‚ 12.296875 disk.rbd2.out‚ 3.033203125" disk.sdb2.out‚1117.42724609375 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 1074.78125 disk.sdb2.in‚3.78125 disk.md0p1.in‚ 12.296875% disk./.usageGB‚2710.6798667907715" disk.sda2.out‚1117.42724609375$ disk.nvme0n1.out‚109198.88671875$ memory_percent‚92.01023578643799 network.sent‚ 2.2096202e+07 network.recv‚ 3.248195e+06) disk./.usagePercent‚75.79174358313173 disk.nvme0n1.in‚ 939.6640625 disk.sdb1.in‚0 disk.dm-0.out‚ 150297.859375* proc.memory.percent‚18.393707275390625# disk.nvme1n1.out‚41098.97265625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sda.in‚8.515625 disk.dm-0.in‚ 1684.23828125Áïžo4×$/ _wandbruntime‚10038  _runtime‚10038zú í`Ø$Š¦ ;"Purity/*FRt*SRqqt3FBt/event_purity‚0.056737590581178665 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15417689085006714 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 *loss/val_total_loss‚40.838314634164426 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9914560317993164 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21556691825389862 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1944444328546524 particle/accuracy_4_of_7‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9444444179534912 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.4375 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 *jet/accuracy_1_of_4‚0.9120569229125977 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5533333420753479 :"Purity/*FRt2SRqqt2FBt/event_purity‚0.03999999910593033 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21251797676086426 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.23078225553035736 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9821428656578064 *jet/accuracy_2_of_6‚0.4615384638309479 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4601978361606598 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5225574374198914 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009924178943037987 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.39542481303215027 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.11999596655368805 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.550000011920929 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26133033633232117 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3882155120372772 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9868854880332947 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.028344672173261642 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.49242833256721497 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.48819881677627563 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0010582010727375746 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11867599189281464 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07456140965223312 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9921690821647644 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4377489387989044 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.016332829371094704 particle/accuracy_2_of_2‚1 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5725258588790894 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.10444444417953491 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14189916849136353 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.48895078897476196 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6111111640930176 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 jet/accuracy_6_of_7‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4108612835407257 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.48735490441322327 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.26086312532424927 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_7‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.23617765307426453 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15462107956409454 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23213790357112885 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.02801724337041378 particle/accuracy_3_of_5‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/event_purity‚0.0714285746216774 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13702130317687988 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_6‚0 particle/accuracy_6_of_6‚1 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7121211886405945 particle/accuracy_3_of_6‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12691675126552582 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5372024178504944 particle/accuracy_5_of_6‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16981036961078644 jet/accuracy_6_of_6‚0 particle/accuracy_1_of_7‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 6 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.124119333922863 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09511525928974152 particle/accuracy_1_of_3‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 particle/accuracy_6_of_7‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.015136713162064552 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.18857085704803467 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3536631762981415 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3509259521961212 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5509259700775146 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.4285714328289032 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5538360476493835 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.48247212171554565 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_2_of_5‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 8"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.466787725687027 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2721264064311981 *jet/accuracy_3_of_5‚0.0771249309182167 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22182025015354156 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13594286143779755 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.41331416368484497 +jet/accuracy_4_of_5‚0.02405914105474949 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21309219300746918 *jet/accuracy_2_of_5‚0.4508768618106842 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.596491277217865 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1495186984539032 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9917411804199219 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4664408564567566 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4985811710357666 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06251799315214157 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ( Purity/1FRt3SRqqt2FBt/FBt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4270833730697632 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06857996433973312 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5208333134651184 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.48307090997695923 ,jet/accuracy_4_of_4‚0.001727357623167336 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9887440204620361 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.40468406677246094 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.24166664481163025 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_3‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.4642857015132904 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 _step‚696 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1994047611951828 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0416666679084301 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.026747064664959908 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 )jet/accuracy_1_of_2‚0.735721230506897 +jet/accuracy_3_of_4‚0.07588043808937073 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8991228342056274 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0885416716337204 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3541666567325592 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.44026121497154236 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.22916637361049652 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.15977013111114502 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 6 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.911458432674408 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5509883165359497 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_6‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1617383062839508 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7058215141296387 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_1‚1 particle/accuracy_7_of_7‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993895292282104 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08099048584699631 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9928160905838013 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9744623899459839 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4691358208656311 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07414640486240387 +jet/accuracy_1_of_1‚0.45146748423576355 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5227472186088562 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9887940287590027 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_6‚0.9807692170143127 particle/accuracy_5_of_7‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5298342704772949 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.46875 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5096085667610168 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.25925928354263306 particle/accuracy_1_of_6‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9990740418434143 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0929383784532547 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.14898991584777832 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4696069359779358 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.37656062841415405 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1125359907746315 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.40573257207870483 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16892756521701813 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9491016864776611 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9899744987487793 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9966490864753723 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5344327688217163 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9985570311546326 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9994331002235413 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 *jet/accuracy_2_of_2‚0.1028657853603363 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9938079118728638 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14229640364646912 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5552816390991211 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,jet/accuracy_3_of_3‚0.019477562978863716 jet/accuracy_5_of_5‚0 % _timestamp‚1.7885059802280524e+09 _runtime‚10038.007639491 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.48739945888519287 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.515162467956543 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14444437623023987 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.025547541677951813 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.38229164481163025 jet/accuracy_7_of_7‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02943476289510727 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4929174780845642 6validation_average_jet_accuracy‚0.4172016382217407 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1754228174686432 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.14846743643283844 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4626753032207489 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 ( Purity/*FRt3SRqqt2FBt/FBt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5541503429412842 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.38257578015327454 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14129634201526642 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.31185415387153625 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_3_of_7‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01344381459057331 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40975379943847656 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4468148350715637 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5478395223617554 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 (particle/accuracy‚0.8528180718421936 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 jet/accuracy_4_of_7‚0 trainer/global_step‚8462 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.49717748165130615 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.39278754591941833 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1666666716337204 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.987550675868988 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4377489387989044 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34513887763023376 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.43975937366485596 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5900189876556396 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt*FžŽV$2Bt/FRt_purity‚0.10000000149011612 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5586195588111877 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0049103377386927605 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.14639639854431152 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_2_of_7‚0.5 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.025625864043831825 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5662744045257568 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.10676850378513336 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.08539094775915146 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 particle/accuracy_1_of_2‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6294347047805786 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9196217060089111 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.13591434061527252 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11026972532272339 *jet/accuracy_1_of_3‚0.8632575273513794 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 'particle/f_score‚0.4136282801628113 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1495186984539032 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3191625475883484 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1954231560230255 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9958333373069763 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3788057863712311 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ( Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.354358047246933 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5074139833450317 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5559210777282715 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0416666679084301 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.44111576676368713 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13927921652793884 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5877424478530884 particle/accuracy_2_of_6‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.016352200880646706 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_7‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.014464925043284893 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.40714287757873535 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5628913044929504 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5387818813323975 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_2_of_4‚0.3728267550468445 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4787071943283081 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.49993205070495605 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5091583728790283 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_5‚1 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993051290512085 +jet/accuracy_2_of_3‚0.24158434569835663 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.37443703413009644 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.28684452176094055 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.48740312457084656 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8039814829826355 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.4166666865348816 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.15380115807056427 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07648809254169464 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0634920671582222 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.474232941865921 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.36667725443840027 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.49748414754867554 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 6 Purity/1FRt3SRqqt*FBt/FBt_purity‚0.970588207244873 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 +jet/accuracy_3_of_6‚0.17307692766189575 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9986525774002075 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.32993414998054504 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16613686084747314 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 *jet/accuracy_1_of_5‚0.9453021883964539 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9899072647094727 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.37402406334877014 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 ,particle/sensitivity‚0.33259040117263794 -"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5625 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3541666567325592 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49666666984558105 epoch‚6 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3673721253871918 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.45325884222984314 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5440971255302429 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.5555555820465088 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.991222083568573 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 jet/accuracy_1_of_7‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17145028710365295 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13919490575790405 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5478671193122864 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5416666269302368 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2605375349521637 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18448303639888763 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0071428571827709675 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4599969983100891 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9946497082710266 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9921364784240723 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1917591691017151 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.46467676758766174 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.26045510172843933 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5|²ÎMÙ$¦ & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22182025015354156 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13594286143779755 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.41331416368484497 +jet/accuracy_4_of_5‚0.02405914105474949 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21309219300746918 *jet/accuracy_2_of_5‚0.4508768618106842 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.596491277217865 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1495186984539032 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9917411804199219 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4664408564567566 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4985811710357666 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06251799315214157 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ( Purity/1FRt3SRqqt2FBt/FBt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4270833730697632 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06857996433973312 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5208333134651184 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.48307090997695923 ,jet/accuracy_4_of_4‚0.001727357623167336 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9887440204620361 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.40468406677246094 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.24166664481163025 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_3‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.4642857015132904 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 _step‚696 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1994047611951828 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0416666679084301 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.026747064664959908 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 )jet/accuracy_1_of_2‚0.735721230506897 +jet/accuracy_3_of_4‚0.07588043808937073 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8991228342056274 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0885416716337204 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3541666567325592 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.44026121497154236 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.22916637361049652 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.15977013111114502 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 6 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.911458432674408 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5509883165359497 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_6‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1617383062839508 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7058215141296387 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_1‚1 particle/accuracy_7_of_7‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993895292282104 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08099048584699631 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9928160905838013 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9744623899459839 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4691358208656311 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07414640486240387 +jet/accuracy_1_of_1‚0.45146748423576355 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5227472186088562 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9887940287590027 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_6‚0.9807692170143127 particle/accuracy_5_of_7‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5298342704772949 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.46875 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5096085667610168 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.25925928354263306 particle/accuracy_1_of_6‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9990740418434143 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0929383784532547 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.14898991584777832 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4696069359779358 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.37656062841415405 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1125359907746315 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.40573257207870483 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16892756521701813 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9491016864776611 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9899744987487793 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9966490864753723 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5344327688217163 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9985570311546326 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9994331002235413 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 *jet/accuracy_2_of_2‚0.1028657853603363 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9938079118728638 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14229640364646912 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5552816390991211 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,jet/accuracy_3_of_3‚0.019477562978863716 jet/accuracy_5_of_5‚0 % _timestamp‚1.7885059802280524e+09 _runtime‚10038.007639491 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.48739945888519287 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.515162467956543 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14444437623023987 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.025547541677951813 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.38229164481163025 jet/accuracy_7_of_7‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02943476289510727 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4929174780845642 6validation_average_jet_accuracy‚0.4172016382217407 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1754228174686432 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.14846743643283844 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4626753032207489 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 ( Purity/*FRt3SRqqt2FBt/FBt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5541503429412842 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.38257578015327454 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14129634201526642 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.31185415387153625 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_3_of_7‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01344381459057331 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40975379943847656 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4468148350715637 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5478395223617554 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 (particle/accuracy‚0.8528180718421936 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 jet/accuracy_4_of_7‚0 trainer/global_step‚8462 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.49717748165130615 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.39278754591941833 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1666666716337204 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.987550675868988 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4377489387989044 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34513887763023376 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.43975937366485596 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5900189876556396 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5586195588111877 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0049103377386927605 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.14639639854431152 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_2_of_7‚0.5 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.025625864043831825 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5662744045257568 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.10676850378513336 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.08539094775915146 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 particle/accuracy_1_of_2‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6294347047805786 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9196217060089111 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.13591434061527252 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11026972532272339 *jet/accuracy_1_of_3‚0.8632575273513794 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 'particle/f_score‚0.4136282801628113 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1495186984539032 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3191625475883484 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1954231560230255 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9958333373069763 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3788057863712311 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ( Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.354358047246933 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5074139833450317 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5559210777282715 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0416666679084301 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.44111576676368713 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13927921652793884 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5877424478530884 particle/accuracy_2_of_6‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.016352200880646706 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_7‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.014464925043284893 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.40714287757873535 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5628913044929504 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5387818813323975 A&Purity/4FRt*SRqqóñÞgHEt*FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_2_of_4‚0.3728267550468445 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4787071943283081 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.49993205070495605 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5091583728790283 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_5‚1 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9993051290512085 +jet/accuracy_2_of_3‚0.24158434569835663 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.37443703413009644 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.28684452176094055 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.48740312457084656 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8039814829826355 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.4166666865348816 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.15380115807056427 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07648809254169464 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0634920671582222 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.474232941865921 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.36667725443840027 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.49748414754867554 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 6 Purity/1FRt3SRqqt*FBt/FBt_purity‚0.970588207244873 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 +jet/accuracy_3_of_6‚0.17307692766189575 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9986525774002075 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.32993414998054504 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16613686084747314 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 *jet/accuracy_1_of_5‚0.9453021883964539 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9899072647094727 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.37402406334877014 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 ,particle/sensitivity‚0.33259040117263794 -"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5625 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3541666567325592 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49666666984558105 epoch‚6 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3673721253871918 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.45325884222984314 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5440971255302429 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.5555555820465088 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.991222083568573 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 jet/accuracy_1_of_7‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17145028710365295 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13919490575790405 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5478671193122864 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5416666269302368 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2605375349521637 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18448303639888763 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0071428571827709675 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4599969983100891 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9946497082710266 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9921364784240723 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1917591691017151 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.46467676758766174 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.26045510172843933 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ;"Purity/*FRt*SRqqt3FBt/event_purity‚0.056737590581178665 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15417689085006714 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 *loss/val_total_loss‚40.838314634164426 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9914560317993164 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21556691825389862 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1944444328546524 particle/accuracy_4_of_7‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9444444179534912 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.4375 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 *jet/accuracy_1_of_4‚0.9120569229125977 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5533333420753479 :"Purity/*FRt2SRqqt2FBt/event_purity‚0.03999999910593033 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21251797676086426 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.23078225553035736 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9821428656578064 *jet/accuracy_2_of_6‚0.4615384638309479 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4601978361606598 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5225574374198914 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009924178943037987 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.39542481303215027 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.11999596655368805 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.550000011920929 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26133033633232117 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3882155120372772 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9868854880332947 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.028344672173261642 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.49242833256721497 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.48819881677627563 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0010582010727375746 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11867599189281464 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07456140965223312 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9921690821647644 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4377489387989044 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.016332829371094704 particle/accuracy_2_of_2‚1 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5725258588790894 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.10444444417953491 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14189916849136353 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.48895078897476196 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6111111640930176 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 jet/accuracy_6_of_7‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4108612835407257 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.48735490441322327 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.26086312532424927 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_7‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.23617765307426453 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15462107956409454 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23213790357112885 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.02801724337041378 particle/accuracy_3_of_5‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/event_purity‚0.0714285746216774 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13702130317687988 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_6‚0 particle/accuracy_6_of_6‚1 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7121211886405945 particle/accuracy_3_of_6‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12691675126552582 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5372024178504944 particle/accuracy_5_of_6‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16981036961078644 jet/accuracy_6_of_6‚0 particle/accuracy_1_of_7‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 6 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.124119333922863 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09511525928974152 particle/accuracy_1_of_3‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 particle/accuracy_6_of_7‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.015136713162064552 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.18857085704803467 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3536631762981415 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3509259521961212 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5509259700775146 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.4285714328289032 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5538360476493835 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.48247212171554565 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_2_of_5‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 8"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.466787725687027 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2721264064311981 *jet/accuracy_3_of_5‚0.0771249309182167 ("Purity/2FRt4SRqqt*FBt/event_purity‚0¸ÎÓá‘Ú$‚(:† ˆßéÔÆ‘Âm gpu.0.gpu‚46.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚150.724' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.681 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\AÈ7jÛ$‚(:ß ˆßéÔº¡–o disk.sda1.out‚0$ memory_percent‚92.00911521911621 disk.dm-0.in‚ 1684.23828125 disk.sdb.out‚1119.0576171875 disk.sda2.in‚8.515625 disk.md0.out‚ 1076.3515625 disk.sda.out‚1119.0576171875 disk.rbd0.in‚0 network.sent‚ 2.2216576e+07 disk.sdb1.in‚0 disk.sdb.in‚3.78125 disk.nvme0n1.in‚ 939.6640625$ disk.nvme0n1.out‚109256.69921875 disk.md0.in‚ 12.296875 disk.rbd2.out‚ 3.033203125 disk.sda1.in‚0! disk.sda2.out‚1119.0576171875 disk.sda.in‚8.515625 disk.rbd0.out‚ 2.74609375( proc.memory.availableMB‚ 1309.2265625 proc.cpu.threads‚17 disk.sdb2.in‚3.78125* proc.memory.percent‚18.393707275390625) disk./.usagePercent‚75.79436839216672 proc.memory.rssMB‚3013.625 network.recv‚ 3.253894e+06% disk./.usageGB‚2710.7737426757812 disk.sdb1.out‚0 cpu‚25.210233754528222 disk.md0p1.in‚ 12.296875 disk.dm-0.out‚ 150356.046875 disk.nvme1n1.in‚ 744.57421875! disk.sdb2.out‚1119.0576171875 disk.md0p1.out‚ 1076.3515625# disk.nvme1n1.out‚41099.34765625 disk.rbd2.in‚0ìàç’Ü$‚(:‡ —ßéÔ´ Èm gpu.0.gpu‚54.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.434' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.8585 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ;¹eÝ$‚(:Ú —ßéÔ˜Àâm cpu‚25.209953345779407 proc.cpu.threads‚17 disk.md0p1.out‚ 1078.1640625 network.recv‚ 3.259046e+06% disk./.usageGB‚2710.8992958068848# disk.nvme1n1.out‚41099.48046875& proc.memory.availableMB‚ 1309.21875 proc.memory.rssMB‚3013.625 disk.nvme0n1.in‚ 939.6640625 disk.rbd2.out‚ 3.033203125 disk.sda.in‚8.515625 disk.sdb.out‚1120.947265625 disk.sdb2.in‚3.78125) disk./.usagePercent‚75.79787891026044 disk.rbd0.in‚0 disk.rbd0.out‚ 2.74609375 disk.dm-0.out‚150356.1796875 disk.sda2.in‚8.515625 disk.md0.out‚ 1078.1640625 disk.rbd2.in‚0$ disk.nvme0n1.out‚109256.69921875 disk.nvme1n1.in‚ 744.57421875 disk.sda2.out‚1120.947265625 disk.sdb2.out‚1120.947265625 disk.sda1.in‚0 disk.sdb1.in‚0 disk.md0p1.in‚ 12.296875 disk.dm-0.in‚ 1684.23828125$ memory_percent‚92.00916290283203* proc.memory.percent‚18.393707275390625 network.sent‚ 2.2329068e+07 disk.md0.in‚ 12.296875 disk.sda.out‚1120.947265625 disk.sda1.out‚0 disk.sdb1.out‚0 disk.sdb.in‚3.78125â>A¾“Þ$‚(:ˆ ¦ßéÔ–µ†l gpu.0.gpu‚25.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚117.193' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.29825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åê«hß$‚(:Ý ¦ßéÔõªm! disk.sdb.out‚1122.44091796875! disk.nvme1n1.out‚ 41329.265625 disk.sdb.in‚3.78125 disk.nvme0n1.in‚ 939.6640625 disk.sda1.out‚0$ memory_percent‚92.01219081878662 proc.cpu.threads‚17$ disk./.usageGB‚2711.148509979248 disk.nvme0n1.out‚ 109487.9375 disk.rbd2.out‚ 3.033203125 disk.md0p1.in‚ 12.296875 disk.dm-0.in‚ 1684.23828125" disk.sda2.out‚1122.44091796875 disk.sda.in‚8.515625 disk.rbd0.in‚0 disk.rbd0.out‚ 2.74609375 cpu‚25.2096936162622 disk.nvme1n1.in‚ 744.57421875 disk.md0.out‚ 1079.58984375 disk.sdb1.out‚0 disk.md0p1.out‚ 1079.58984375 network.recv‚ 3.263776e+06) disk./.usagePercent‚75.80484704282395 disk.md0.in‚ 12.296875! disk.sda.out‚1122.44091796875 disk.sda1.in‚0 disk.sdb1.in‚0) proc.memory.availableMB‚ 1308.72265625* proc.memory.percent‚18.393707275390625 network.sent‚ 2.2333185e+07 disk.sda2.in‚8.515625 disk.dm-0.out‚ 150817.203125 disk.rbd2.in‚0 disk.sdb2.in‚3.78125" disk.sdb2.out‚1122.44091796875 proc.memory.rssMB‚3013.625hŽ…4à$/ _wandbruntime‚10082  _runtime‚10082Šg:Êá$Ä ' lr-AdamW/pg1‚0.00025836635692606065 ' lr-AdamW/pg2‚0.00025836635692606065 trainer/global_step‚8499 % _timestamp‚1.7885060248476455e+09 _runtime‚10082.625086785 _step‚697ƒûœÆÏâ$É _runtime‚10082.625086785 _step‚697 ' lr-AdamW/pg1‚0.00025836635692606065 ' lr-AdamW/pg2‚0.00025836635692606065 trainer/global_step‚8499 % _timestamp‚1.7885060248476455e+09¹œœ·4ã$/ _wandbruntime‚10082  _runtime‚10082·µgTv&ä$ûO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51701658 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 106776392 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 215523357 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 178195708 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚727466694793216 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 393745411 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚791753493408256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 106776390 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9751548496384 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 215523357 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚8499 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚717715146296832 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 178195708 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 215549701 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚801935973693952 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9751548496384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 178195710 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚717715146296832 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚727465766818816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚630 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚703623487634607 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚693909300625968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9714187008639 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10182243115008 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 _runtime‚10082.626939394 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 393719065 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 _step‚698 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚727466694793216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚717715128293888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9750638524928 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237170688 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚727466694793216 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 393745411 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚791753493408256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 106776390 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9751548496384 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 215523357 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚8499 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚717715146296832 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 178195708 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 215549701 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚801935973693952 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9751548496384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 178195710 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚717715146296832 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚727465766818816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚630 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚703623487634607 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚693909300625968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9714187008639 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10182243115008 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 _runtime‚10082.626939394 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 393719065 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 _step‚698 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚727466694793216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚717715128293888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9750638524928 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237170688 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 215523357 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 178195708 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚717796765382656 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51706582 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 178215868 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚727548507364864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 393790428 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9715304281917 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 _runtime‚10083.842821403 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 215574558 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 215574558 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚703703622502429 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚717796747379712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9752669969920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚791843719112704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51707206 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 215548188 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 106788455 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158495037 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚693988335259872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 393764056 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚626 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 178215870 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10183651092480 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 215548188 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚727549435352576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9716212257129 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9752669969920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 393764056 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 178215870 _step‚699 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10183422323712 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚693988318220512 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚717796747379712 % _timestamp‚1.7885060260648932e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 106788457 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚802027370205184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚727549435352576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚717796765382656 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640óª'è$ŠN I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚717796747379712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9752669969920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚791843719112704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51707206 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 215548188 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 106788455 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158495037 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚693988335259872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 393764056 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚626 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 178215870 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10183651092480 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 215548188 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚727549435352576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9716212257129 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9752669969920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 393764056 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 178215870 _step‚699 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10183422323712 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚693988318220512 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚717796747379712 % _timestamp‚1.7885060260648932e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 106788457 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚802027370205184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚727549435352576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚717796765382656 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚717796765382656 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51706582 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 178215868 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚727548507364864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 393790428 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9715304281917 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 _runtime‚10083.842821403 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 215574558 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 215574558 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚703703622502429».M$Í4é$/ _wandbruntime‚10083  _runtime‚10083­*¿°_ê$Ù 0loss/FBt3/assignment_loss‚1.0399443113581381 /loss/FRt1/detection_loss‚1.7980686381188842 .loss/FRt2/detection_loss‚2.126068829298019 0loss/FRt1/assignment_loss‚2.2972824490347574 0loss/FRt4/assignment_loss‚2.8171512427784147 0loss/SRqqt1/detection_loss‚2.265820966412624 1loss/SRqqt2/detection_loss‚2.5897565061847367 /loss/FBt1/detection_loss‚0.5468260631530679 .loss/FBt3/detection_loss‚1.977808654308319 /loss/FBt4/detection_loss‚2.1564349063805173 epoch‚7 2loss/SRqqt1/assignment_loss‚1.3789287662754457 1loss/SRqqt4/detection_loss‚2.0249196688334146 /loss/FBt2/detection_loss‚2.0268018676175013 _runtime‚10083.844218399 1loss/SRqqt3/detection_loss‚0.8606145617599457 &loss/total_loss‚39.521330781844455 /loss/FRt3/assignment_loss‚1.297767589489619 1loss/FBt1/assignment_loss‚0.08280503698581144 trainer/global_step‚8499 2loss/SRqqt4/assignment_loss‚1.4158881573062954 2loss/SRqqt3/assignment_loss‚0.7435142978694353 1loss/FBt2/assignment_loss‚0.21862324803361036 1loss/FBt4/assignment_loss‚0.29234874056298094 /loss/FRt4/detection_loss‚3.0720975285484675 _step‚700 2loss/SRqqt2/assignment_loss‚1.7664755571944017 /loss/FRt3/detection_loss‚1.7083762582667448 % _timestamp‚1.7885060260670907e+09 /loss/FRt2/assignment_loss‚3.017006936073303jlbzUë$Þ 1loss/SRqqt3/detection_loss‚0.8606145617599457 &loss/total_loss‚39.521330781844455 /loss/FRt3/assignment_loss‚1.297767589489619 1loss/FBt1/assignment_loss‚0.08280503698581144 trainer/global_step‚8499 2loss/SRqqt4/assignment_loss‚1.4158881573062954 2loss/SRqqt3/assignment_loss‚0.7435142978694353 1loss/FBt2/assignment_loss‚0.21862324803361036 1loss/FBt4/assignment_loss‚0.29234874056298094 /loss/FRt4/detection_loss‚3.0720975285484675 _step‚700 2loss/SRqqt2/assignment_loss‚1.7664755571944017 /loss/FRt3/detection_loss‚1.708376258266744dèçj8 % _timestamp‚1.7885060260670907e+09 /loss/FRt2/assignment_loss‚3.017006936073303 0loss/FBt3/assignment_loss‚1.0399443113581381 /loss/FRt1/detection_loss‚1.7980686381188842 .loss/FRt2/detection_loss‚2.126068829298019 0loss/FRt1/assignment_loss‚2.2972824490347574 0loss/FRt4/assignment_loss‚2.8171512427784147 0loss/SRqqt1/detection_loss‚2.265820966412624 1loss/SRqqt2/detection_loss‚2.5897565061847367 /loss/FBt1/detection_loss‚0.5468260631530679 .loss/FBt3/detection_loss‚1.977808654308319 /loss/FBt4/detection_loss‚2.1564349063805173 epoch‚7 2loss/SRqqt1/assignment_loss‚1.3789287662754457 1loss/SRqqt4/detection_loss‚2.0249196688334146 /loss/FBt2/detection_loss‚2.0268018676175013 _runtime‚10083.844218399¼”Jéfì$‚(:Û µßéÔܯþm disk.sdb2.in‚3.78125 disk.sda1.in‚0 disk.rbd0.out‚ 2.74609375 proc.cpu.threads‚17 disk.rbd2.out‚ 3.033203125 disk.md0p1.in‚ 12.296875 disk.sdb.in‚3.78125 disk.sdb.out‚1124.1376953125! disk.sda2.out‚1124.1376953125 disk.nvme1n1.in‚ 744.57421875 disk.md0.out‚ 1081.234375! disk.sdb2.out‚1124.1376953125 disk.sda.in‚8.515625* proc.memory.percent‚18.393707275390625% disk./.usageGB‚2711.1484909057617 disk.nvme0n1.in‚ 939.6640625 disk.nvme0n1.out‚ 109487.9375 disk.md0.in‚ 12.296875 disk.sda.out‚1124.1376953125 disk.sda1.out‚0 disk.rbd0.in‚0 proc.memory.rssMB‚3013.625# disk.nvme1n1.out‚41329.48828125 cpu‚25.209428666998015 network.sent‚ 2.2350459e+07 disk.sdb1.out‚0 disk.md0p1.out‚ 1081.234375 disk.dm-0.in‚ 1684.23828125! disk.dm-0.out‚150817.42578125$ memory_percent‚92.00921058654785 disk.rbd2.in‚0 disk.sdb1.in‚0( proc.memory.availableMB‚ 1309.2109375 network.recv‚ 3.266624e+06) disk./.usagePercent‚75.80484650952128 disk.sda2.in‚8.515625Yû'’í$‚(:‡ µßéÔÛ¤ñm gpu.0.gpu‚12.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.386' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.5965 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CaWœî$‚(:‘ ÄßéÔ¶ÂÕk gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚117.81' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.452499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö‡m¯lï$‚(:á ÄßéÔ˜ìœn disk.md0.in‚ 12.296875) disk./.usagePercent‚75.80484650952128 disk.rbd0.out‚ 2.74609375! disk.dm-0.out‚151162.93359375 disk.sdb.in‚3.78125" disk.sda2.out‚1125.49560546875 disk.sdb1.out‚0 disk.nvme0n1.out‚ 109661.25! disk.sda.out‚1125.49560546875 disk.md0.out‚ 1082.54296875 disk.rbd2.in‚0 disk.sda1.in‚0 disk.md0p1.out‚ 1082.54296875 disk.sda2.in‚8.515625# memory_percent‚92.0093297958374 disk.nvme1n1.in‚ 744.57421875 network.sent‚ 2.2447921e+07 disk.sda.in‚8.515625 disk.dm-0.in‚ 1684.23828125) proc.memory.availableMB‚ 1309.19140625 disk.sdb2.in‚3.78125" disk.sdb2.out‚1125.49560546875 disk.sda1.out‚0 disk.rbd0.in‚0! disk.sdb.out‚1125.49560546875 disk.nvme0n1.in‚ 939.6640625 cpu‚25.209191471068845 proc.cpu.threads‚17 network.recv‚ 3.271844e+06 disk.sdb1.in‚0 disk.md0p1.in‚ 12.296875# disk.nvme1n1.out‚41501.68359375 disk.rbd2.out‚ 3.033203125% disk./.usageGB‚2711.1484909057617 proc.memory.rssMB‚3013.625* proc.memory.percent‚18.393707275390625¢²"Óð$‚(:ˆ ÓßéÔûËŠm gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.813' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.70325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02=\kñ$‚(:à ÓßéÔœäâm# disk.nvme0n1.out‚109747.8984375 disk.nvme0n1.in‚ 939.6640625 disk.sdb2.in‚3.78125 disk.sdb1.out‚0) disk./.usagePercent‚75.79010293082663% disk./.usageGB‚2710.6211891174316 disk.md0p1.in‚ 12.296875! disk.sdb.out‚1127.41943359375" disk.sda2.out‚1127.41943359375 disk.md0.in‚ 12.296875 disk.md0p1.out‚ 1084.40234375 disk.sdb.in‚3.78125" disk.nvme1n1.out‚ 41616.5390625 network.recv‚ 3.276053e+06 disk.md0.out‚ 1084.40234375 disk.sda1.in‚0 disk.rbd0.out‚ 2.74609375 disk.sda2.in‚8.515625$ memory_percent‚92.00918674468994* proc.memory.percent‚18.393707275390625 disk.rbd2.in‚0 disk.rbd2.out‚ 3.033203125 disk.sda.in‚8.515625 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1684.23828125 cpu‚25.20893393386178" disk.sdb2.out‚1127.41943359375! disk.sda.out‚1127.41943359375 disk.dm-0.out‚ 151364.4375 proc.memory.rssMB‚3013.625 proc.cpu.threads‚17 network.sent‚ 2.245163e+07 disk.sdb1.in‚0 disk.nvme1n1.in‚ 744.57421875) proc.memory.availableMB‚ 1309.21484375—²“ò$‚(:ˆ âßéÔ¸¤²m gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.891' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.72275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iÅžhó$‚(:Ý âßéÔ²áËm disk.rbd2.in‚0 disk.rbd2.out‚ 3.033203125 disk.sdb2.in‚3.78125 disk.sda1.in‚0 cpu‚25.208651153077806 disk.rbd0.in‚0 disk.md0p1.out‚ 1086.06640625$ memory_percent‚92.00997352600098( proc.memory.availableMB‚ 1309.0859375* proc.memory.percent‚18.393707275390625 proc.cpu.threads‚17 disk.rbd0.out‚ 2.74609375 disk.sdb1.out‚0 disk.md0.in‚ 12.296875 disk.sda.in‚8.515625 disk.dm-0.in‚ 1684.23828125 disk.sdb.in‚3.78125 network.recv‚ 3.280724e+06! disk.dm-0.out‚151529.17578125 disk.sdb.out‚1129.162109375 disk.nvme1n1.in‚ 744.57421875 disk.sda.out‚1129.162109375) disk./.usagePercent‚75.78869447849996 disk.md0p1.in‚ 12.296875 disk.sda2.in‚8.515625# disk.nvme0n1.out‚109796.7890625# disk.nvme1n1.out‚41732.38671875 disk.md0.out‚ 1086.06640625 network.sent‚ 2.2459585e+07$ disk./.usageGB‚2710.570816040039 disk.sdb2.out‚1129.162109375 disk.sda1.out‚0 proc.memory.rssMB‚3013.625 disk.sdb1.in‚0 disk.sda2.out‚1129.162109375 disk.nvme0n1.in‚ 939.6640625º„S¸4ô$/ _wandbruntime‚10143  _runtime‚10143AaÇõ$Á $ _timestamp‚1.788506086008964e+09 _runtime‚10143.786329812 _step‚701 & lr-AdamW/pg1‚0.0002518560240277071 & lr-AdamW/pg2‚0.0002518560240277071 trainer/global_step‚8549I4Æ Ìö$Æ _runtime‚10143.786329812 _step‚701 & lr-AdamW/pg1‚0.0002518560240277071 & lr-AdamW/pg2‚0.0002518560240277071 trainer/global_step‚8549 $ _timestamp‚1.788506086008964e+09½N–a=4÷$/ _wandbruntime‚10143  _runtime‚10143éq½(ø$ûO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚796264754690560 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9807635921408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚731602808535040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚630 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 107379640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚731602808535040 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚731603736509440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚721796082585088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚806505396464640 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚707631169631279 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 216794603 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 216768259 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚628 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚721796082585088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚796264778630656 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 % _timestamp‚1.7885060860106366e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 179203708 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚697860180353168 _step‚702 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51986219 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚731603736509440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 216768259 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9770064263747 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 trainer/global_step‚8549 _runtime‚10143.788266642 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9806725949952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9770972238751 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 179203708 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9807635921408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 179203710 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51985591 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10240393246208 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51986219 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚731603736509440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 216768259 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9770064263747 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 trainer/global_step‚8549 _runtime‚10143.788266642 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9806725949952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9770972238751 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 179203708 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9807635921408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 179203710 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51985591 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10240393246208 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚796264754690560 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9807635921408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚731602808535040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚630 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 107379640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚731602808535040 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚731603736509440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚721796082585088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚806505396464640 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚707631169631279 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 216794603 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 216768259 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚628 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚721796082585088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚796264778630656 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 % _timestamp‚1.7885060860106366e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 179203708 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚697860180353168 _step‚702 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚697939214987072 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚10144.987491556 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚731686477068800 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 107391707 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 396016958 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚8549 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚731686477068800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 159383590 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 396043330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚635 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10241543089664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚703 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9772089512237 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚697939197947712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9807847410176 SÀ°¬÷¿?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚707710379484737 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚721877719673856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 396016958 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 $ _timestamp‚1.788506087208476e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚731685549081088 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 216793090 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 216793090 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚806596523484672 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51991883 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 396043330 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 216819460 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚806596784582144 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159382955 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51991250 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 179223870 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 216819460 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 179223868 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚633 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570¡ø+i'ü$‰N L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 396043330 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 216819460 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚806596784582144 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159382955 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51991250 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 179223870 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 216819460 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 179223868 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚633 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚721877719673856 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 179223870 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚721877701670912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚796354980395008 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 107391705 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 179223868 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9808757394944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚697939214987072 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚10144.987491556 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚731686477068800 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 107391707 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 396016958 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚8549 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚731686477068800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 159383590 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 396043330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚635 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10241543089664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚703 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9772089512237 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚697939197947712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9807847410176 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚707710379484737 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚721877719673856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 396016958 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 $ _timestamp‚1.788506087208476e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚731685549081088 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 216793090 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 216793090 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚806596523484672 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51991883 Bp“„%‚(:ˆ àéÔáÛäk gpu.0.gpu‚36.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.763' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.94075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ!i…%‚(:Þ àéÔêàžm disk.sdb1.in‚0 disk.dm-0.in‚ 1684.23828125" disk.sda2.out‚1134.23681640625 disk.nvme0n1.in‚ 939.6640625 proc.cpu.threads‚17 disk.rbd0.out‚ 2.74609375 disk.md0p1.out‚ 1090.953125! disk.sdb.out‚1134.23681640625 disk.nvme1n1.in‚ 744.57421875" disk.sdb2.out‚1134.23681640625# disk.nvme1n1.out‚42142.06640625* proc.memory.percent‚18.393707275390625 cpu‚25.20788031966161 disk.sda2.in‚8.515625 disk.sdb2.in‚3.78125! disk.sda.out‚1134.23681640625 proc.memory.rssMB‚3013.625) disk./.usagePercent‚75.79437212528534 disk.rbd2.out‚ 3.033203125 disk.sda1.in‚0( proc.memory.availableMB‚ 1308.3359375 network.sent‚ 2.2578185e+07 network.recv‚ 3.2936e+06% disk./.usageGB‚2710.7738761901855" disk.nvme0n1.out‚ 109796.921875 disk.md0.in‚ 12.296875 disk.md0.out‚ 1090.953125 disk.rbd0.in‚0 disk.md0p1.in‚ 12.296875 disk.sdb.in‚3.78125 disk.rbd2.in‚0 disk.sda.in‚8.515625 disk.sdb1.out‚0! disk.dm-0.out‚151938.98828125 disk.sda1.out‚0$ memory_percent‚92.01455116271973¢¯‹Ón†%‚(:ã žàéÔÊ ƒm disk.md0.out‚ 1092.859375) disk./.usagePercent‚75.79781448729904 disk.rbd2.in‚0 disk.rbd2.out‚ 3.033203125) proc.memory.availableMB‚ 1308.58984375 cpu‚25.207625372605797 disk.sdb2.in‚3.78125 disk.sdb1.in‚0 disk.sda2.in‚8.515625 disk.nvme1n1.in‚ 744.57421875 disk.sda.in‚8.515625 disk.sdb1.out‚0 disk.sdb.in‚3.78125! disk.sdb.out‚1136.22802734375$ memory_percent‚92.01300144195557% disk./.usageGB‚2710.8969917297363 disk.sda1.in‚0 disk.md0p1.out‚ 1092.859375! disk.dm-0.out‚151994.26953125* proc.memory.percent‚18.393707275390625 network.sent‚ 2.2582361e+07 disk.rbd0.in‚0 disk.md0.in‚ 12.296875 proc.memory.rssMB‚3013.625 proc.cpu.threads‚17 network.recv‚ 3.298464e+06" disk.sdb2.out‚1136.22802734375 disk.md0p1.in‚ 12.296875 disk.nvme0n1.in‚ 939.6640625# disk.nvme0n1.out‚109797.1484375! disk.sda.out‚1136.22802734375 disk.rbd0.out‚ 2.74609375# disk.nvme1n1.out‚42197.12109375 disk.sda1.out‚0 disk.dm-0.in‚ 1684.23828125" disk.sda2.out‚1136.228027343759óÀ3“‡%‚(:ˆ žàéÔÚm gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.219' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.55475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*¨Å4ˆ%/ _wandbruntime‚10205  _runtime‚10205y›Ø›É‰%à trainer/global_step‚8599 % _timestamp‚1.7885061474280531e+09 _runtime‚10205.20549731 _step‚705 ' lr-AdamW/pg1‚0.00024540285918201056 ' lr-AdamW/pg2‚0.00024540285918201056ç—ô7Ί%È _step‚705 ' lr-AdamW/pg1‚0.00024540285918201056 ' lr-AdamW/pg2‚0.00024540285918201056 trainer/global_step‚8599 % _timestamp‚1.7885061474280531e+09 _runtime‚10205.20549731ÁÞº÷ú4‹%/ _wandbruntime‚10205  _runtime‚10205+¤åÌâŒ%ûO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 218039505 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 180211710 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218013161 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _runtime‚10205.207669739 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 160254287 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9863723346432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚800776039912960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚701811060080368 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚725877054879232 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9862813374976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚725877036876288 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚725877036876288 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9826849493859 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 398224869 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚735740778225664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚701811077119728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 trainer/global_step‚8599 % _timestamp‚1.7885061474299264e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10298787203584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚735739850251264 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52271397 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 _step‚706 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 398251215 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 180211708 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚725877054879232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bÓñóØ ytes.small_pool.peak‚ 1189768192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 218013161 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚735739850251264 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52271397 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 _step‚706 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 398251215 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 180211708 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚725877054879232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 218013161 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218013161 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _runtime‚10205.207669739 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 160254287 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9863723346432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚800776039912960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚701811060080368 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚725877054879232 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9862813374976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚725877036876288 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚725877036876288 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9826849493859 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 398224869 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚735740778225664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚701811077119728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640ÂÚXÇL4Ž%/ _wandbruntime‚10206  _runtime‚10206Æ/ŽÀ '%…N K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 180231868 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 160272739 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 _step‚707 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 180231868 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10299948771840 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 107994955 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10300181734912 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚617 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 180231870 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 218037992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚735823518785024 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 160272122 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚811166214389248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚701890094714272 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9864844819968 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 398269860 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚711718061481617 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚735823518785024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚725958673965056 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 218064362 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7885061485533488e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 398296232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 218037992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚800866289557504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9827966767345 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 218064362 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9863934835200 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 107994957 _runtime‚10206.331263134 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚735822590797312 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚711717136467045 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚615 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ÏKt'%ŠN E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 218037992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚735823518785024 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 160272122 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚811166214389248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚701890094714272 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9864844819968 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 398269860 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚711718061481617 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚735823518785024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚725958673965056 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 218064362 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7885061485533488e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 398296232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 218037992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚800866289557504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9827966767345 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 218064362 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9863934835200 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 107994957 _runtime‚10206.331263134 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚735822590797312 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚711717136467045 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚615 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 180231868 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 160272739 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 _step‚707 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 180231868 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10299948771840 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 107994955 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10300181734912 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚617 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 180231870 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521ÃÆ Z¥4‘%/ _wandbruntime‚10206  _runtime‚10206ƒ²a|'’%Ï /loss/FRt3/detection_loss‚1.õ¼>B.7250979227828793 0loss/SRqqt3/detection_loss‚0.824951997765946 /loss/FBt1/detection_loss‚0.6824467810234283 trainer/global_step‚8599 0loss/FRt3/assignment_loss‚1.3327742107212543 2loss/SRqqt3/assignment_loss‚0.9047127281914332 .loss/FRt4/detection_loss‚3.322883211904102 1loss/SRqqt1/detection_loss‚2.4825244212851807 .loss/FBt3/detection_loss‚2.749987632036209 -loss/FBt2/detection_loss‚2.37285415828228 /loss/FRt1/assignment_loss‚2.294312281811491 2loss/SRqqt4/assignment_loss‚1.4506460381671786 0loss/SRqqt2/detection_loss‚2.967064837614695 1loss/SRqqt4/detection_loss‚2.5258707279960313 .loss/FBt4/detection_loss‚2.548896476626396 2loss/SRqqt2/assignment_loss‚1.1065971197353468 1loss/FBt2/assignment_loss‚0.24906645821204165 0loss/FBt4/assignment_loss‚0.2621060673029234 # _timestamp‚1.78850614855531e+09 _step‚708 /loss/FRt1/detection_loss‚1.6339689314828572 2loss/SRqqt1/assignment_loss‚2.0417253620484295 1loss/FBt1/assignment_loss‚0.15629706716585934 %loss/total_loss‚42.97153812569883 _runtime‚10206.33265728 .loss/FRt2/detection_loss‚2.300921634691102 /loss/FRt2/assignment_loss‚3.098168671131134 /loss/FRt4/assignment_loss‚3.699378410975138 1loss/FBt3/assignment_loss‚0.23828497674548998 epoch‚7eϼ„Z“%Ô /loss/FRt3/detection_loss‚1.7250979227828793 0loss/SRqqt3/detection_loss‚0.824951997765946 /loss/FBt1/detection_loss‚0.6824467810234283 trainer/global_step‚8599 0loss/FRt3/assignment_loss‚1.3327742107212543 2loss/SRqqt3/assignment_loss‚0.9047127281914332 .loss/FRt4/detection_loss‚3.322883211904102 1loss/SRqqt1/detection_loss‚2.4825244212851807 .loss/FBt3/detection_loss‚2.749987632036209 -loss/FBt2/detection_loss‚2.37285415828228 /loss/FRt1/assignment_loss‚2.294312281811491 2loss/SRqqt4/assignment_loss‚1.4506460381671786 0loss/SRqqt2/detection_loss‚2.967064837614695 1loss/SRqqt4/detection_loss‚2.5258707279960313 .loss/FBt4/detection_loss‚2.548896476626396 2loss/SRqqt2/assignment_loss‚1.1065971197353468 1loss/FBt2/assignment_loss‚0.24906645821204165 0loss/FBt4/assignment_loss‚0.2621060673029234 # _timestamp‚1.78850614855531e+09 _step‚708 /loss/FRt1/detection_loss‚1.6339689314828572 2loss/SRqqt1/assignment_loss‚2.0417253620484295 1loss/FBt1/assignment_loss‚0.15629706716585934 %loss/total_loss‚42.97153812569883 _runtime‚10206.33265728 .loss/FRt2/detection_loss‚2.300921634691102 /loss/FRt2/assignment_loss‚3.098168671131134 /loss/FRt4/assignment_loss‚3.699378410975138 1loss/FBt3/assignment_loss‚0.23828497674548998 epoch‚7Äï]ˆ’’”%‚(:‡ ­àéÔ‚×úl gpu.0.gpu‚23.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚178.914' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.7285 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êÏ„,k•%‚(:à ­àéÔºãm! disk.nvme1n1.out‚ 42502.421875* proc.memory.percent‚18.393707275390625 proc.cpu.threads‚17# disk.nvme0n1.out‚109797.1484375 disk.sdb1.in‚0 disk.md0.in‚ 12.296875 disk.md0.out‚ 1094.4765625 proc.memory.rssMB‚3013.625! disk.sda.out‚1137.90771484375 disk.sda1.in‚0 disk.dm-0.out‚152299.5703125! disk.sdb.out‚1137.90771484375 network.recv‚ 3.301781e+06 disk.nvme1n1.in‚ 744.57421875 disk.rbd2.out‚ 3.033203125 disk.sdb2.in‚3.78125" disk.sdb2.out‚1137.90771484375 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚ 12.296875( proc.memory.availableMB‚ 1308.3203125 network.sent‚ 2.2600457e+07 disk.rbd0.out‚ 2.74609375 disk.sdb.in‚3.78125 disk.nvme0n1.in‚ 939.6640625 disk.rbd2.in‚0$ memory_percent‚92.01464653015137 cpu‚25.207341779062727) disk./.usagePercent‚75.80067810926543 disk.dm-0.in‚ 1684.23828125$ disk./.usageGB‚2710.999408721924 disk.sda.in‚8.515625 disk.md0p1.out‚ 1094.4765625 disk.sda2.in‚8.515625" disk.sda2.out‚1137.90771484375Eh–%‚(:’ ¼àéÔË„™m gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬¹{dn—%‚(:ã ¼àéÔ¼Ãÿm% disk./.usageGB‚2710.0936698913574 disk.sda1.in‚0 disk.md0p1.in‚ 12.296875! disk.dm-0.out‚152476.43359375 disk.nvme0n1.in‚ 939.6640625 disk.rbd0.out‚ 2.74609375 disk.sdb.in‚3.78125 proc.cpu.threads‚17) disk./.usagePercent‚75.77535327248165 disk.sdb1.out‚0 disk.md0p1.out‚ 1096.3671875# disk.nvme1n1.out‚42679.23046875 network.sent‚ 2.2695234e+07 disk.sda.in‚8.515625 disk.dm-0.in‚ 1684.23828125 disk.nvme1n1.in‚ 744.57421875 disk.rbd2.out‚ 3.033203125 proc.memory.rssMB‚3013.625 disk.sda2.in‚8.515625 disk.sdb2.in‚3.78125! disk.sda.out‚1139.86767578125 disk.sda1.out‚0 disk.rbd0.in‚0 disk.md0.out‚ 1096.3671875 disk.md0.in‚ 12.296875 network.recv‚ 3.307451e+06" disk.sdb2.out‚1139.86767578125 disk.sdb1.in‚0" disk.nvme0n1.out‚ 109797.203125 disk.rbd2.in‚0 cpu‚25.20708612214951! disk.sdb.out‚1139.86767578125" disk.sda2.out‚1139.86767578125$ memory_percent‚92.01462268829346) proc.memory.availableMB‚ 1308.32421875* proc.memory.percent‚18.393707275390625Å‘˜%‚(:† ËàéÔ¼ƒöl gpu.0.gpu‚54.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.512' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zßï ^™%‚(:Ó ËàéÔÏÝám disk.nvme1n1.in‚ 744.57421875) proc.memory.availableMB‚ 1308.56640625 disk.md0.in‚ 12.296875 disk.sdb2.in‚3.78125 network.sent‚ 2.2703255e+07 network.recv‚ 3.311354e+06 disk.sda2.out‚ 1141.73828125 disk.sda1.in‚0 disk.md0p1.in‚ 12.296875 disk.dm-0.in‚ 1684.23828125" disk.nvme0n1.out‚ 109797.203125 disk.sda.out‚ 1141.73828125 disk.rbd0.out‚ 2.74609375 proc.memory.rssMB‚3013.625 disk.sdb1.in‚0 disk.md0p1.out‚ 1098.171875! disk.dm-0.out‚152627.46484375 disk.sdb.out‚ 1141.73828125# disk.nvme1n1.out‚42830.26171875 disk.rbd2.in‚0 disk.sda1.out‚0 cpu‚25.20680923246616 disk.sdb.in‚3.78125 disk.sda2.in‚8.515625 disk.rbd2.out‚ 3.033203125 disk.sda.in‚8.515625$ memory_percent‚92.01314449310303 proc.cpu.threads‚17) disk./.usagePercent‚75.77532586072489 disk.sdb1.out‚0 disk.md0.out‚ 1098.171875 disk.rbd0.in‚0* proc.memory.percent‚18.393707275390625 disk.sdb2.out‚ 1141.73828125# disk./.usageGB‚2710.09268951416 disk.nvme0n1.in‚ 939.6640625j˜Ôñ4š%/ _wandbruntime‚10256  _runtime‚10256x–îžÇ›%Á & lr-AdamW/pg1‚0.0002390082072188527 & lr-AdamW/pg2‚0.0002390082072188527 trainer/global_step‚8649 $ _timestamp‚1.788506198488745e+09 _runtime‚10256.266140221 _step‚709a×[Ìœ%Æ _runtime‚10256.266140221 _step‚709 & lr-AdamW/pg1‚0.0002390082072188527 & lr-AdamW/pg2‚0.0002390082072188527 trainer/global_step‚8649 $ _timestamp‚1.788506198488745e+09ÅPHÿê4%/ _wandbruntime‚10256  _runtime‚10256WÉON(ž%ûO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52558771 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 _runtime‚10256.266914365 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚739876891967488 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9881818773963 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 181219710 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 219258063 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10357083395584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚815644669641728 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10357320566272 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 108586140 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚739877819941888 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚805287349075456 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 161144250 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚739877819941888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚729957991167488 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 _step‚710 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚729958009170432 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 400504117 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 219258063 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9919810771456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52558110 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9919810771456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚661 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 400477771 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 108586142 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 181219710 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚739876891967488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 219284407 % _timestamp‚1.7885061984895325e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9918900800000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 219284407 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9882726748967 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚715644683595895 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚815644408530944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚729958009170432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚705761956846928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚705761939807568 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 181219708 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚805287325135360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 161144913 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0òY=(Ÿ%€P N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 161144250 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚739877819941888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚729957991167488 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 _step‚710 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚729958009170432 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 400504117 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 219258063 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9919810771456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52558110 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9919810771456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚661 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 400477771 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 108586142 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 181219710 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚739876891967488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 219284407 % _timestamp‚1.7885061984895325e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9918900800000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 219284407 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9882726748967 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚715644683595895 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚815644408530944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚729958009170432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚705761956846928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚705761939807568 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 181219708 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚805287325135360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 161144913 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52558771 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 _runtime‚10256.266914365 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚739876891967488 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9881818773963 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 181219710 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 219258063 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10357083395584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚815644669641728 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10357320566272 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 108586140 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚739877819941888 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚805287349075456 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364ÆÕK[4 %/ _wandbruntime‚10257  _runtime‚10257<ÚYÛ¡%…N XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚815735789065728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚711 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 400549134 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 219309264 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚730039628256256 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚705840974441472 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52563885 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 219282894 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚739960560501248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚739959632513536 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚730039610253312 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚715724818463925 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10358475383296 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9882936047241 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚805377574779904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 181239870 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9920022260224 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 181239870 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9920022260224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚739960560501248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 trainer/global_step‚8649 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 H@DeviceStatsMonitor.on_train_batc`!0h_end/inactive_split.all.current‚661 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 108598207 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52564544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚10257.294517191 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 161162090 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚730039628256256 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9883844022453 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 219282894 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 400549134 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7885061995147805e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚730039610253312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 181239868 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚6593D­J'¢%ŠN ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚730039628256256 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9883844022453 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 219282894 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 400549134 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7885061995147805e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚730039610253312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 181239868 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚659 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚815735789065728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚711 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 400549134 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 219309264 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚730039628256256 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚705840974441472 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52563885 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 219282894 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚739960560501248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚739959632513536 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚730039610253312 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚715724818463925 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10358475383296 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9882936047241 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚805377574779904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 181239870 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9920022260224 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 181239870 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9920022260224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚739960560501248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 trainer/global_step‚8649 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚661 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 108598207 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52564544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚10257.294517191 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 161162090 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185Ç!Yi 4£%/ _wandbruntime‚10257  _runtime‚10257jE uZ¤%Ô /loss/FRt4/detection_loss‚3.0894293189048767 % _timestamp‚1.7885061995190215e+09 /loss/FRt1/detection_loss‚1.7507816587616285 0loss/SRqqt2/detection_loss‚2.460243965570743 2loss/SRqqt3/assignment_loss‚0.8779503281852654 1loss/FBt3/assignment_loss‚0.11934594701354702 .loss/FBt3/detection_loss‚2.328018546104431 trainer/global_step‚8649 _runtime‚10257.29642073 _step‚712 0loss/FRt2/assignment_loss‚3.6076070208882176 2loss/SRqqt4/assignment_loss‚1.6119713919237255 1loss/FBt2/assignment_loss‚0.23624763928038353 1loss/SRqqt3/detection_loss‚0.7481793120051875 epoch‚7 .loss/FRt2/detection_loss‚2.413372524819055 1loss/SRqqt4/detection_loss‚2.6909807920455933 %loss/total_loss‚41.30924829122417 0loss/SRqqt1/assignment_loss‚1.50056591629982 .loss/FRt3/detection_loss‚1.587602822172145 1loss/SRqqt1/detection_loss‚2.2369085114735823 /loss/FBt1/detection_loss‚0.6578672964272922 /loss/FRt1/assignment_loss‚2.463184970346364 /loss/FRt3/assignment_loss‚1.378875873486201 0loss/FRt4/assignment_loss‚3.2340713491042457 2loss/SRqqt2/assignment_loss‚1.3280709744072878 1loss/FBt1/assignment_loss‚0.15703015457878608 1loss/FBt4/assignment_loss‚0.16663402117434695 /loss/FBt2/detection_loss‚2.1388800564934227 .loss/FBt4/detection_loss‚2.5254278997580212;çA_¥%Ù 1loss/FBt3/assignment_loss‚0.11934594701354702 .loss/FBt3/detection_loss‚2.328018546104431 trainer/global_step‚8649 _runtime‚10257.29642073 _step‚712 0loss/FRt2/assignment_loss‚3.6076070208882176 2loss/SRqqt4/assignment_loss‚1.6119713919237255 1loss/FBt2/assignment_loss‚0.23624763928038353 1loss/SRqqt3/detection_loss‚0.7481793120051875 epoch‚7 .loss/FRt2/detection_loss‚2.413372524819055 1loss/SRqqt4/detection_loss‚2.6909807920455933 %loss/total_loss‚41.30924829122417 0loss/SRqqt1/assignment_loss‚1.50056591629982 .loss/FRt3/detection_loss‚1.587602822172145 1loss/SRqqt1/detection_loss‚2.2369085114735823 /loss/FBt1/detection_loss‚0.6578672964272922 /loss/FRt1/assignment_loss‚2.463184970346364 /loss/FRt3/assignment_loss‚1.378875873486201 0loss/FRt4/assignment_loss‚3.2340713491042457 2loss/SRqqt2/assignment_loss‚1.3280709744072878 1loss/FBt1/assignment_loss‚0.15703015457878608 1loss/FBt4/assignment_loss‚0.16663402117434695 /loss/FBt2/detection_loss‚2.1388800564934227 .loss/FBt4/detection_loss‚2.525427899758021 /loss/FRt4/detection_loss‚3.0894293189048767 % _timestamp‚1.7885061995190215e+09 /loss/FRt1/detection_loss‚1.7507816587616285 0loss/SRqqt2/detection_loss‚2.460243965570743 2loss/SRqqt3/assignment_loss‚0.8779503281852654Èßté¦%‚(:’ ÚàéÔ‘åm gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚126.497' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚31.624249999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0C±h§%‚(:Ý ÚàéÔÖµ¯o! disk.sdb2.out‚1143.2978515625 network.sent‚ 2.2720168e+07 disk.nvme1n1.in‚ 744.57421875 disk.md0.in‚ 12.296875 disk.sda1.out‚0 disk.sdb1.in‚0 disk.md0p1.out‚ 1099.65625 disk.sda2.in‚8.515625) disk./.usagePercent‚75.76058036214066 disk.nvme0n1.in‚ 939.6640625 disk.sda.in‚8.515625 disk.sda.out‚1143.2978515625 disk.rbd0.out‚ 2.74609375 disk.sdb2.in‚3.78125 network.recv‚ 3.316087e+06# disk.nvme0n1.out‚109797.2734375# disk.nvme1n1.out‚42918.66796875! disk.dm-0.out‚152715.94140625 disk.sdb.in‚3.78125* proc.memory.percent‚18.393707275390625 disk.md0.out‚ 1099.65625 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚ 12.296875 proc.memory.rssMB‚3013.625 disk.sda1.in‚0 disk.dm-0.in‚ 1684.23828125 disk.sdb.out‚1143.2978515625! disk.sda2.out‚1143.2978515625 disk.rbd2.in‚0 disk.rbd2.out‚ 3.033203125$ memory_percent‚92.01762676239014 cpu‚25.206523703084144 proc.cpu.threads‚17% disk./.usageGB‚2709.5653190612793) proc.memory.availableMB‚ 1307.83203125ã[9òg¨%‚(:Ü éàéÔÎÄàl) disk./.usagePercent‚75.75780569505805$ memory_percent‚92.01664924621582 disk.sdb.out‚1145.447265625* proc.memory.percent‚18.393707275390625% disk./.usageGB‚2709.4660835266113 disk.sdb.in‚3.78125 disk.sda2.out‚1145.447265625# disk.nvme0n1.out‚109797.2734375 proc.memory.rssMB‚3013.625 network.sent‚ 2.2816464e+07 disk.sdb1.out‚0 disk.dm-0.in‚ 1684.23828125 disk.sda2.in‚8.515625 disk.nvme1n1.in‚ 744.57421875# disk.nvme1n1.out‚42968.49609375 disk.rbd2.in‚0 cpu‚25.206284851687474 disk.md0.out‚ 1101.7265625 disk.sdb2.in‚3.78125 network.recv‚ 3.320714e+06 disk.md0p1.in‚ 12.296875 disk.nvme0n1.in‚ 939.6640625 disk.rbd2.out‚ 3.033203125 disk.sda.in‚8.515625 disk.sda.out‚1145.447265625 disk.sda1.in‚0( proc.memory.availableMB‚ 1307.9921875 disk.sda1.out‚0 disk.md0p1.out‚ 1101.7265625! disk.dm-0.out‚152765.76953125 disk.md0.in‚ 12.296875 disk.sdb2.out‚1145.447265625 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 2.74609375:šy“©%‚(:ˆ éàéÔ¸ý€m gpu.0.gpu‚18.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.953' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.48825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œÎä¹’ª%‚(:‡ øàéÔþγk gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.206' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.0515 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z°ýn«%‚(:ã øàéÔØþél disk.rbd2.out‚ 3.033203125! disk.sdb.out‚1146.86767578125 disk.rbd2.in‚0 disk.rbd0.out‚ 2.74609375 disk.dm-0.in‚ 1684.23828125# memory_percent‚92.0180082321167) disk./.usagePercent‚75.76278652858707" disk.sda2.out‚1146.86767578125" disk.nvme0n1.out‚ 109952.609375" disk.sdb2.out‚1146.86767578125 disk.dm-0.out‚152922.0859375) proc.memory.availableMB‚ 1307.76953125 proc.memory.rssMB‚3013.625* proc.memory.percent‚18.393707275390625 disk.sdb1.in‚0 network.sent‚ 2.2820406e+07 disk.nvme1n1.in‚ 744.57421875" disk.nvme1n1.out‚ 42969.4765625 disk.md0.out‚ 1103.08203125 disk.sda2.in‚8.515625 disk.sda.in‚8.515625 disk.md0p1.out‚ 1103.08203125 proc.cpu.threads‚17! disk.sda.out‚1146.86767578125 disk.sda1.out‚0 disk.rbd0.in‚0 disk.md0.in‚ 12.296875 disk.sdb2.in‚3.78125 disk.sdb1.out‚0 disk.md0p1.in‚ 12.296875% disk./.usageGB‚2709.6442222595215 disk.sdb.in‚3.78125 disk.nvme0n1.in‚ 939.6640625 disk.sda1.in‚0 cpu‚25.206008729996743 network.recv‚ 3.325261e+06¿b^m¬%‚(:â ‡áéÔ™³¿m" disk.nvme1n1.out‚ 42969.6328125) disk./.usagePercent‚75.76284348531122 cpu‚25.20575460771998 network.sent‚ 2.2826207e+07 disk.sda.in‚8.515625 disk.dm-0.in‚ 1684.23828125 disk.dm-0.out‚153002.7890625 disk.nvme1n1.in‚ 744.57421875 proc.cpu.threads‚17 disk.rbd0.out‚ 2.74609375! disk.sdb.out‚1148.36376953125 disk.rbd2.in‚0 disk.rbd2.out‚ 3.033203125$ memory_percent‚92.01653003692627 disk.sdb2.in‚3.78125" disk.sdb2.out‚1148.36376953125 disk.sda1.out‚0 disk.md0p1.in‚ 12.296875 disk.nvme0n1.in‚ 939.6640625 disk.md0.in‚ 12.296875) proc.memory.availableMB‚ 1308.01171875 disk.rbd0.in‚0 disk.md0.out‚ 1104.51171875 proc.memory.rssMB‚3013.625 network.recv‚ 3.329155e+06 disk.sdb1.in‚0" disk.sda2.out‚1148.36376953125! disk.nvme0n1.out‚ 110033.15625! disk.sda.out‚1148.36376953125 disk.md0p1.out‚ 1104.51171875 disk.sdb.in‚3.78125* proc.memory.percent‚18.393707275390625% disk./.usageGB‚2709.6462593078613 disk.sda1.in‚0 disk.sdb1.out‚0 disk.sda2.in‚8.515625&2 –“­%‚(:ˆ ‡áéÔãám gpu.0.gpu‚22.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚142.653' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.66325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìN9æ4®%/ _wandbruntime‚10312  _runtime‚10312Ãn¡Ê¯%Ä trainer/global_step‚8699 % _timestamp‚1.7885062546710415e+09 _runtime‚10312.448233473 _step‚713 ' lr-AdamW/pg1‚0.00023267340077411832 ' lr-AdamW/pg2‚0.00023267340077411832^4LÏ°%É % _timestamp‚1.7885062546710415e+09 _runtime‚10312.448233473 _step‚713 ' lr-AdamW/pg1‚0.00023267340077411832 ' lr-AdamW/pg2‚0.00023267340077411832 trainer/global_step‚8699Éð¤4±%/ _wandbruntime‚10312  _runtime‚10312UŲXU²%úO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚709712819534768 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10415712933376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚719651440578203 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52846245 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚714 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚744014861658112 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9974988225024 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚709712836574128 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 109189390 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚643 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 220529309 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 162035635 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9975898196480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9937696029071 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 220529309 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚645 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 182227708 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 182227708 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9975898196480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _runtime‚10312.449290018 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 402730673 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 182227710 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚734038945458688 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 182227710 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚820214347231232 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 162036280 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚820214084023296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 109189392 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚744013933683712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚734038963461632 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚734038963461632 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚719650515563839 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9974988225024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52846888 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚809798634297856 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 trainer/global_step‚8699 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚744013933683712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚734038963461632 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚734038963461632 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚719650515563839 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9974988225024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52846888 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚809798634297856 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 trainer/global_step‚8699 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10415712933376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚719651440578203 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52846245 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚714 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚744014861658112 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9974988225024 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚709712836574128 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 109189390 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚643 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 220529309 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 162035635 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9975898196480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9937696029071 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 220529309 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚645 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 182227708 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 182227708 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9975898196480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _runtime‚10312.449290018 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 402730673 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 182227710 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚734038945458688 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 182227710 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚820214347231232 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 162036280 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚820214084023296Ê¥ž^³4´%/ _wandbruntime‚10313  _runtime‚10313¤åxY 'µ%…N OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 220527796 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚744096674229760 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚744097602217472 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚616 % _timestamp‚1.7885062557084332e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 402802036 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚820305743517184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚809888836062208 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10416671523328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚809888860002304 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 402775664 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9976109685248 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 220527796 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 162053951 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚734120582547456 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 109201457 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 109201455 _runtime‚10313.486255279 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚719731575446233 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚709791837129312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9976109685248 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 220554166 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚734120564544512 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 trainer/global_step‚8699 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9938813302349 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 182247868 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 211991552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 182247870 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚734120564544512 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162053333 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9977019670016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52851878 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 182247870 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚820305507585536 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10416883514880 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 _step‚715 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 402802036 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 402775664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚709791854168672 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52852494 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856]õš†î%¶%ŠN RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162053333 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9977019670016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52851878 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 182247870 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚820305507585536 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10416883514880 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 _step‚715 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 402802036 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 402775664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚709791854168672 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52852494 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 220527796 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚744096674229760 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚744097602217472 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚616 % _timestamp‚1.7885062557084332e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 402802036 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚820305743517184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚809888836062208 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10416671523328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚809888860002304 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 402775664 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9976109685248 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 220527796 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 162053951 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚734120582547456 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 109201457 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 109201455 _runtime‚10313.486255279 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚719731575446233 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚709791837129312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9976109685248 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 220554166 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚734120564544512 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 trainer/global_step‚8699 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9938813302349 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 182247868 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 211991552 F@DeviceStatsMonitor.on_train_batch_end/ove>Ћ„"rsize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 182247870 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚734120564544512ËQŒl64·%/ _wandbruntime‚10313  _runtime‚10313 Á¨x[¸%Õ 2loss/SRqqt3/assignment_loss‚0.9177994131002852 /loss/FRt1/detection_loss‚1.5889615383810856 .loss/FRt4/detection_loss‚3.090546170106301 1loss/SRqqt4/detection_loss‚2.4489120168070637 _step‚716 1loss/SRqqt2/assignment_loss‚1.560293557588011 0loss/FRt2/assignment_loss‚3.0783795306557105 /loss/FBt1/detection_loss‚0.6176638676440471 % _timestamp‚1.7885062557102206e+09 0loss/FRt1/assignment_loss‚1.9752898931503295 1loss/FBt1/assignment_loss‚0.10720614853844163 /loss/FBt3/assignment_loss‚0.071285050262564 1loss/SRqqt3/detection_loss‚0.8672541830965803 1loss/FBt4/assignment_loss‚0.20502658965230391 .loss/FRt3/detection_loss‚1.655256428216633 /loss/FRt4/assignment_loss‚2.843595972427955 1loss/SRqqt1/assignment_loss‚1.394554472973813 /loss/FRt2/detection_loss‚2.1231138040883497 1loss/SRqqt2/detection_loss‚2.2658994048833847 .loss/FBt4/detection_loss‚2.058118595679601 epoch‚7 1loss/FBt2/assignment_loss‚0.23495144621219052 1loss/SRqqt1/detection_loss‚2.4455878060796987 .loss/FBt2/detection_loss‚2.281716747717424 %loss/total_loss‚38.74901802425976 trainer/global_step‚8699 0loss/FRt3/assignment_loss‚1.2460357588634157 2loss/SRqqt4/assignment_loss‚1.2646468279342498 /loss/FBt3/detection_loss‚2.4069228002003262 _runtime‚10313.487262674B”—`¹%Ú trainer/global_step‚8699 0loss/FRt3/assignment_loss‚1.2460357588634157 2loss/SRqqt4/assignment_loss‚1.2646468279342498 /loss/FBt3/detection_loss‚2.4069228002003262 _runtime‚10313.487262674 2loss/SRqqt3/assignment_loss‚0.9177994131002852 /loss/FRt1/detection_loss‚1.5889615383810856 .loss/FRt4/detection_loss‚3.090546170106301 1loss/SRqqt4/detection_loss‚2.4489120168070637 _step‚716 1loss/SRqqt2/assignment_loss‚1.560293557588011 0loss/FRt2/assignment_loss‚3.0783795306557105 /loss/FBt1/detection_loss‚0.6176638676440471 % _timestamp‚1.7885062557102206e+09 0loss/FRt1/assignment_loss‚1.9752898931503295 1loss/FBt1/assignment_loss‚0.10720614853844163 /loss/FBt3/assignment_loss‚0.071285050262564 1loss/SRqqt3/detection_loss‚0.8672541830965803 1loss/FBt4/assignment_loss‚0.20502658965230391 .loss/FRt3/detection_loss‚1.655256428216633 /loss/FRt4/assignment_loss‚2.843595972427955 1loss/SRqqt1/assignment_loss‚1.394554472973813 /loss/FRt2/detection_loss‚2.1231138040883497 1loss/SRqqt2/detection_loss‚2.2658994048833847 .loss/FBt4/detection_loss‚2.058118595679601 epoch‚7 1loss/FBt2/assignment_loss‚0.23495144621219052 1loss/SRqqt1/detection_loss‚2.4455878060796987 .loss/FBt2/detection_loss‚2.281716747717424 %loss/total_loss‚38.74901802425976Ì–õ á’º%‚(:‡ –áéÔÇ ôl gpu.0.gpu‚6.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úÂã·h»%‚(:Ý –áéÔÙ¬m# disk.nvme1n1.out‚42970.08984375 disk.md0.in‚ 12.296875 disk.md0.out‚ 1106.4296875 disk.rbd2.in‚0 disk.rbd0.out‚ 2.74609375 disk.sdb1.out‚0 disk.sda2.in‚8.515625* proc.memory.percent‚18.393707275390625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.in‚ 12.296875 disk.sdb.in‚3.78125" disk.sda2.out‚1150.35595703125 disk.sda.in‚8.515625$ proc.memory.availableMB‚1307.625 cpu‚25.205505451915126 disk.dm-0.in‚ 1684.23828125 disk.dm-0.out‚ 153032.21875 disk.sdb2.in‚3.78125 disk.rbd0.in‚0) disk./.usagePercent‚75.76647932953018$ disk./.usageGB‚2709.776294708252 proc.memory.rssMB‚3013.625 network.recv‚ 3.333937e+06 disk.sdb1.in‚0$ memory_percent‚92.01889038085938 network.sent‚ 2.2832487e+07 disk.md0p1.out‚ 1106.4296875" disk.sdb2.out‚1150.35595703125! disk.sda.out‚1150.35595703125! disk.sdb.out‚1150.35595703125$ disk.nvme0n1.out‚110062.12890625 disk.rbd2.out‚ 3.033203125 disk.nvme0n1.in‚ 939.6640625 disk.nvme1n1.in‚ 744.57421875Å?Aב¼%‚(:† ¥áéÔÚì˜m gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚159.696' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.924 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓKc½%‚(:Ø ¥áéÔⶄn disk.nvme1n1.in‚ 744.57421875$ disk./.usageGB‚2709.963634490967 disk.dm-0.in‚ 1684.23828125% proc.memory.availableMB‚ 1307.8125* proc.memory.percent‚18.393707275390625 disk.sdb.out‚1151.904296875# disk.nvme0n1.out‚110240.0859375 disk.rbd2.in‚0 disk.sdb2.in‚3.78125 disk.sda.in‚8.515625 disk.sda.out‚1151.904296875 cpu‚25.20525259213821 disk.rbd2.out‚ 3.033203125 disk.rbd0.out‚ 2.74609375 disk.sda2.in‚8.515625 network.sent‚ 2.293937e+07 network.recv‚ 3.338333e+06 disk.sdb1.out‚0 proc.memory.rssMB‚3013.625 proc.cpu.threads‚17 disk.nvme0n1.in‚ 939.6640625 disk.md0.in‚ 12.296875 disk.sdb2.out‚1151.904296875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sdb1.in‚0$ memory_percent‚92.01774597167969 disk.md0.out‚ 1107.92578125 disk.sda1.in‚0) disk./.usagePercent‚75.77171742826269 disk.md0p1.in‚ 12.296875 disk.md0p1.out‚ 1107.92578125! disk.dm-0.out‚153210.27734375# disk.nvme1n1.out‚42970.19140625 disk.sdb.in‚3.78125 disk.sda2.out‚1151.904296875”E;Ñd¾%‚(:Ù ´áéÔáÄÛl disk.md0p1.in‚ 12.296875$ memory_percent‚92.01927185058594$ disk./.usageGB‚2710.276077270508 disk.sda2.in‚8.515625! disk.sda2.out‚1153.4013671875 disk.sdb1.in‚0* proc.memory.percent‚18.393707275390625 disk.sda.out‚1153.4013671875 disk.sda1.out‚0 disk.md0p1.out‚ 1109.359375 disk.dm-0.in‚ 1684.23828125 proc.memory.rssMB‚3013.625 cpu‚25.20498665266053! disk.dm-0.out‚153348.29296875 disk.sdb.in‚3.78125% proc.memory.availableMB‚ 1307.5625 proc.cpu.threads‚17) disk./.usagePercent‚75.78045345914612 disk.nvme0n1.in‚ 939.6640625$ disk.nvme0n1.out‚110377.73828125 disk.md0.out‚ 1109.359375 disk.sda.in‚8.515625 disk.sda1.in‚0 disk.rbd2.in‚0 disk.md0.in‚ 12.296875 disk.rbd2.out‚ 3.033203125 disk.sdb.out‚1153.4013671875 network.sent‚ 2.2947569e+07 disk.nvme1n1.in‚ 744.57421875! disk.sdb2.out‚1153.4013671875 disk.rbd0.in‚0 disk.rbd0.out‚ 2.74609375 network.recv‚ 3.342668e+06" disk.nvme1n1.out‚ 42970.5546875 disk.sdb2.in‚3.78125 disk.sdb1.out‚0·ßåC“¿%‚(:ˆ ´áéÔŽÎël gpu.0.gpu‚20.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.129' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.03225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°ªkÀ%‚(:à ÃáéÔØ­†m# disk.nvme1n1.out‚42970.73046875) disk./.usagePercent‚75.78394104522543 disk.sdb2.in‚3.78125 disk.sda.in‚8.515625! disk.sda.out‚1155.34228515625 disk.rbd0.in‚0 disk.sdb1.in‚0" disk.sdb2.out‚1155.34228515625 disk.md0p1.out‚ 1111.234375 disk.nvme0n1.in‚ 939.6640625 cpu‚25.204713093277597 network.sent‚ 2.2950999e+07 disk.sda2.in‚8.515625 disk.nvme1n1.in‚ 744.57421875 disk.md0.out‚ 1111.234375 disk.rbd0.out‚ 2.74609375 disk.dm-0.out‚ 153469.65625 disk.rbd2.in‚0 network.recv‚ 3.345933e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.in‚ 12.296875" disk.sda2.out‚1155.34228515625$ memory_percent‚92.01929569244385) proc.memory.availableMB‚ 1307.55859375 disk.sdb.in‚3.78125! disk.sdb.out‚1155.34228515625 disk.rbd2.out‚ 3.033203125 proc.memory.rssMB‚3013.625* proc.memory.percent‚18.393707275390625 proc.cpu.threads‚17$ disk./.usageGB‚2710.400810241699 disk.dm-0.in‚ 1684.23828125$ disk.nvme0n1.out‚110498.92578125 disk.md0.in‚ 12.296875 disk.sdb1.out‚0©÷ (“Á%‚(:ˆ ÃáéÔ¯å‘m gpu.0.gpu‚35.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.481' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.62025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãöñÇ4Â%/ _wandbruntime‚10372  _runtime‚10372ÔÄHÉÃ%à _step‚717 ' lr-AdamW/pg1‚0.00022639976001197532 ' lr-AdamW/pg2‚0.00022639976001197532 trainer/global_step‚8749 $ _timestamp‚1.788506315062856e+09 _runtime‚10372.840092316M¸ ÚÎÄ%È _step‚717 ' lr-AdamW/pg1‚0.00022639976001197532 ' lr-AdamW/pg2‚0.00022639976001197532 trainer/global_step‚8749 $ _timestamp‚1.788506315062856e+09 _runtime‚10372.840092316ÍÙ&ÚÜ4Å%/ _wandbruntime‚10372  _runtime‚10372^Ûžu(Æ%ÿO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 183235710 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10031985621504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚10372.841575673 _step‚718 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 183235708 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚824783724294656 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚738119899749888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚814309895580160 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10474057496576 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 221747867 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 221747867 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 162927690 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚824783977016832 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚723657272546147 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 162927040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚748151903374336 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚748150975399936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚748151903374336 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 109792640 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚713663699261968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚650 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 405009921 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 109792642 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 % _timestamp‚1.7885063150640156e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚738119899749888 trainer/global_step‚8749 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚814309919520256 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚713663716301328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 221774211 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53135048 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9993573284179 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚738119917752832 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 183235708 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9994481259183 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚723658197560511 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 404983575 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10031985621504 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 221774211 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 183235710 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0ÛiÁ (Ç%„P P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 221747867 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 221747867 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 162927690 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚824783977016832 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚723657272546147 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 162927040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚748151903374336 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚748150975399936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚748151903374336 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 109792640 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚713663699261968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚650 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 405009921 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 109792642 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 % _timestamp‚1.7885063150640156e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚738119899749888 trainer/global_step‚8749 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚814309919520256 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚713663716301328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 221774211 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53135048 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9993573284179 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚738119917752832 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 183235708 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9994481259183 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚723658197560511 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 404983575 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10031985621504 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 221774211 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 183235710 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 183235710 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10031985621504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚10372.841575673 _step‚718 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 183235708 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚824783724294656 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚738119899749888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚814309895580160 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10474057496576 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725Ϊ4áp4È%/ _wandbruntime‚10374  _runtime‚10374é<›> É%‰N OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 183255868 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10032197110272 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 trainer/global_step‚8749 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 183255870 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚738201536838656 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 221772698 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162944904 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53140821 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚824875363386368 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚738201518835712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 405054938 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚723737407413969 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚719 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10033107095040 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚738201536838656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7885063162762406e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 162945528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9994690557457 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 _runtime‚10374.055628399 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚748234643933696 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 221799068 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 109804705 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53140199 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 221799068 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 405028566 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 183255868 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚738201518835712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚748234643933696 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚723738332428541 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10033107095040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235060224ÂnF'Ê%ŽN XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9994690557457 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 _runtime‚10374.055628399 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚748234643933696 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 221799068 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 109804705 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53140199 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 221799068 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 405028566 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 183255868 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚738201518835712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚748234643933696 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚723738332428541 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10033107095040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235060224 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 183255868 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10032197110272 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 trainer/global_step‚8749 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 183255870 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚738201536838656 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 221772698 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162944904 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53140821 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚824875363386368 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚738201518835712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 405054938 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚723737407413969 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚719 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10033107095040 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚738201536838656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7885063162762406e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 162945528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0Ï^&Óõ4Ë%/ _wandbruntime‚10374  _runtime‚10374˜uo`Ì%Ú 0loss/FRt4/assignment_loss‚2.8381830900907516 2loss/SRqqt3/assignment_loss‚0.9593402217776417 1loss/FBt1/assignment_loss‚0.04272178650920555 1loss/SRqqt3/detection_loss‚0.9298976746395351 0loss/FRt1/assignment_loss‚1.9817626517872478 2loss/SRqqt1/assignment_loss‚1.2861920176073909 1loss/SRqqt2/detection_loss‚2.8721127639646116 0loss/SRqqt4/detection_loss‚2.212535930176576 1loss/FBt2/assignment_loss‚0.18580662054221517 0loss/FBt3/assignment_loss‚0.2696245727720452 1loss/FBt4/assignment_loss‚0.45031659325940154 .loss/FRt4/detection_loss‚2.997930599110467 &loss/total_loss‚39.145000237024036 trainer/global_step‚8749 2loss/SRqqt2/assignment_loss‚1.1302168553938037 _runtime‚10374.057099836 0loss/FRt2/assignment_loss‚3.0664534607659215 /loss/FBt1/detection_loss‚0.5745207166582986 /loss/FBt4/detection_loss‚2.6291380375623703 $ _timestamp‚1.788506316279878e+09 /loss/FBt3/detection_loss‚1.8014987744390965 /loss/FRt1/detection_loss‚1.8625193757580683 .loss/FRt2/detection_loss‚2.173808315039977 1loss/SRqqt1/detection_loss‚2.4376659082869687 /loss/FBt2/detection_loss‚1.7922028473445348 0loss/FRt3/assignment_loss‚1.2292660667346074 /loss/FRt3/detection_loss‚1.7788106835805453 epoch‚7 2loss/SRqqt4/assignment_loss‚1.6424746732227504 _step‚720¹Ck±eÍ%ß 0loss/FRt4/assignment_loss‚2.8381830900907516 2loss/SRqqt3/assignment_loss‚0.9593402217776417 1loss/FBt1/assignment_loss‚0.04272178650920555 1loss/SRqqt3/detection_loss‚0.9298976746395351 0loss/FRt1/assignment_loss‚1.9817626517872478 2loss/SRqqt1/assignment_loss‚1.2861920176073909 1loss/SRqqt2/detection_loss‚2.8721127639646116 0loss/SRqqt4/detection_loss‚2.212535930176576 1loss/FBt2/assignment_loss‚0.18580662054221517 0loss/FBt3/assignment_loss‚0.2696245727720452 1loss/FBt4/assignment_loss‚0.45031659325940154 .loss/FRt4/detection_loss‚2.997930599110467 &loss/total_loss‚39.145000237024036 trainer/global_step‚8749 2loss/SRqqt2/assignment_loss‚1.1302168553938037 _runtime‚10374.057099836 0loss/FRt2/assignment_loss‚3.0664534607659215 /loss/FBt1/detection_loss‚0.5745207166582986 /loss/FBt4/detection_loss‚2.6291380375623703 $ _timestamp‚1.788506316279878e+09 /loss/FBt3/detection_loss‚1.8014987744390965 /loss/FRt1/detection_loss‚1.8625193757580683 .loss/FRt2/detection_loss‚2.173808315039977 1loss/SRqqt1/detection_loss‚2.4376659082869687 /loss/FBt2/detection_loss‚1.7922028473445348 0loss/FRt3/assignment_loss‚1.2292660667346074 /loss/FRt3/detection_loss‚1.7788106835805453 epoch‚7 2loss/SRqqt4/assignment_loss‚1.6424746732227504 _step‚720ÐÏïª.“Î%‚(:ˆ ÒáéÔ¥ÄÚl gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚133.349' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.33725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?RhnÏ%‚(:ã ÒáéÔõn) disk./.usagePercent‚75.79095962822063$ disk./.usageGB‚2710.651828765869$ disk.nvme0n1.out‚110499.34765625 disk.rbd2.out‚ 3.033203125 disk.sda.in‚8.515625 disk.rbd0.in‚0 disk.nvme1n1.out‚ 42971.46875 network.recv‚ 3.350537e+06" disk.sdb2.out‚1156.79345703125 disk.rbd0.out‚ 2.74609375 disk.dm-0.in‚ 1684.23828125! disk.sdb.out‚1156.79345703125 disk.sda2.in‚8.515625 disk.md0.in‚ 12.296875$ memory_percent‚92.01931953430176 disk.sdb2.in‚3.78125 disk.sda1.out‚0 disk.nvme0n1.in‚ 939.6640625 disk.rbd2.in‚0( proc.memory.availableMB‚ 1307.5546875* proc.memory.percent‚18.393707275390625 cpu‚25.204485202686364 proc.cpu.threads‚17 disk.md0p1.in‚ 12.296875" disk.sda2.out‚1156.79345703125 disk.md0.out‚ 1112.63671875 network.sent‚ 2.2969264e+07 disk.sdb1.out‚0 disk.sdb.in‚3.78125 disk.nvme1n1.in‚ 744.57421875 disk.sda1.in‚0 disk.sdb1.in‚0 disk.md0p1.out‚ 1112.63671875! disk.sda.out‚1156.79345703125! disk.dm-0.out‚153470.81640625 proc.memory.rssMB‚3013.625“9“Ð%‚(:ˆ ááéÔ¨Ýñk gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.481' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.62025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åY¬MpÑ%‚(:å ááéÔ™¶¼m disk.sda1.out‚0# disk.nvme0n1.out‚110988.3984375 disk.md0.in‚ 12.296875$ memory_percent‚92.01953411102295 proc.memory.rssMB‚3013.625 network.sent‚ 2.3063604e+07 network.recv‚ 3.355752e+06( disk./.usagePercent‚75.7761885311087! disk.sdb.out‚1158.27294921875 disk.nvme1n1.in‚ 744.57421875" disk.sdb2.out‚1158.27294921875 disk.sda.in‚8.515625 cpu‚25.20424155353804 disk.md0p1.in‚ 12.296875 disk.sdb2.in‚3.78125 proc.cpu.threads‚17% disk./.usageGB‚2710.1235427856445 disk.rbd0.in‚0! disk.dm-0.out‚153960.05859375 disk.sda2.in‚8.515625# disk.nvme1n1.out‚42971.66015625* proc.memory.percent‚18.393707275390625 disk.sda1.in‚0 disk.sdb1.in‚0 disk.rbd0.out‚ 2.74609375 disk.sdb1.out‚0! disk.sda.out‚1158.27294921875 disk.sdb.in‚3.78125" disk.sda2.out‚1158.27294921875 disk.rbd2.in‚0 disk.md0p1.out‚ 1114.04296875 disk.dm-0.in‚ 1684.23828125 disk.nvme0n1.in‚ 939.6640625 disk.md0.out‚ 1114.04296875 disk.rbd2.out‚ 3.033203125) proc.memory.availableMB‚ 1307.51953125Âìo‚’Ò%‚(:‡ ðáéÔ÷¿¿m gpu.0.gpu‚41.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚156.294' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.0735 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖyhÓ%‚(:Ý ðáéÔ’’¥n) disk./.usagePercent‚75.79022164399889 disk.nvme1n1.in‚ 744.57421875 disk.sda.in‚8.515625 disk.md0p1.out‚ 1115.8515625 proc.cpu.threads‚17 disk.dm-0.in‚ 1684.23828125 disk.sdb2.in‚3.78125 proc.memory.rssMB‚3013.625 disk.sdb1.out‚0 disk.sdb.out‚1160.1298828125 disk.sda.out‚1160.1298828125 disk.sda1.out‚0 cpu‚25.20396778838809 disk.md0.in‚ 12.296875$ memory_percent‚92.01645851135254! disk.sda2.out‚1160.1298828125$ disk.nvme0n1.out‚110988.48046875 disk.md0.out‚ 1115.8515625 disk.rbd2.in‚0! disk.sdb2.out‚1160.1298828125 disk.rbd0.in‚0! disk.dm-0.out‚153960.38671875 disk.sda1.in‚0 disk.rbd0.out‚ 2.74609375 disk.sdb1.in‚0 disk.md0p1.in‚ 12.296875 disk.nvme1n1.out‚ 42971.90625 network.sent‚ 2.3071753e+07% disk./.usageGB‚2710.6254348754883 disk.sda2.in‚8.515625 disk.rbd2.out‚ 3.033203125( proc.memory.availableMB‚ 1308.0234375 network.recv‚ 3.360216e+06 disk.sdb.in‚3.78125 disk.nvme0n1.in‚ 939.6640625* proc.memory.percent‚18.393707275390625UIY‘Ô%‚(:† ÿáéÔžÊøl gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.344' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µí6YÕ%‚(:Î ÿáéԪĪm disk.sda1.in‚0 disk.rbd0.in‚0 disk.md0p1.out‚ 1117.40625 disk.sda2.out‚ 1161.74609375! disk.nvme0n1.out‚ 111103.78125$ memory_percent‚92.01650619506836 network.recv‚ 3.36437e+06 disk.sda.in‚8.515625 disk.sdb1.out‚0 disk.sdb.in‚3.78125 proc.memory.rssMB‚3013.625 disk.nvme1n1.in‚ 744.57421875* proc.memory.percent‚18.393707275390625 cpu‚25.20372766073502) disk./.usagePercent‚75.79086683355769 disk.md0p1.in‚ 12.296875 disk.sdb.out‚ 1161.74609375 disk.nvme0n1.in‚ 939.6640625 proc.cpu.threads‚17 network.sent‚ 2.3077618e+07# disk.nvme1n1.out‚43049.12890625 disk.sdb1.in‚0! disk.dm-0.out‚154152.91015625 disk.sda2.in‚8.515625$ disk./.usageGB‚2710.648509979248 disk.md0.in‚ 12.296875 disk.rbd2.in‚0 disk.sdb2.in‚3.78125 disk.sdb2.out‚ 1161.74609375 disk.rbd0.out‚ 2.74609375 disk.sda.out‚ 1161.74609375 disk.sda1.out‚0 disk.dm-0.in‚ 1684.23828125' proc.memory.availableMB‚ 1308.015625 disk.md0.out‚ 1117.40625 disk.rbd2.out‚ 3.033203125ñ^úï4Ö%/ _wandbruntime‚10433  _runtime‚10433Ëë=È×% _step‚721 & lr-AdamW/pg1‚0.0002201885923497555 & lr-AdamW/pg2‚0.0002201885923497555 trainer/global_step‚8799 % _timestamp‚1.7885063758311255e+09 _runtime‚10433.610738079«îÍØ%Ç trainer/global_step‚8799 % _timestamp‚1.7885063758311255e+09 _runtime‚10433.610738079 _step‚721 & lr-AdamW/pg1‚0.0002201885923497555 & lr-AdamW/pg2‚0.0002201885923497555Ñ “4Ù%/ _wandbruntime‚10433  _runtime‚10433®Æí²¿Ú%€P H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 222992769 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 110395892 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚818821180802560 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚727664954542819 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚829353507029504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚717614578989168 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 184243710 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10088073046528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10532326226944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10050358514291 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10088073046528 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 184243710 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚752288017116160 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚742200854041088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 184243708 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10049450539287 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53424006 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 163819259 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10532555009024 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _step‚722 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚742200872044032 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 $ _timestamp‚1.788506375835455e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 $ _timestamp‚1.788506375835455e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 222992769 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 110395892 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚818821180802560 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚727664954542819 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚829353507029504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚717614578989168 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 184243710 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10088073046528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10532326226944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10050358514291 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10088073046528 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 184243710 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚752288017116160 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚742200854041088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 184243708 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10049450539287 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53424006 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 163819259 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10532555009024 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _step‚722 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚742200872044032 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚8799 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚631 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 163837769 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚717693613623072 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 223043970 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚742282473126912 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚818911406507008 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚717693596583712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚742282491129856 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚629 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚727744164396277 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283008 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10050567812565 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 223017600 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 184263868 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚752371685649920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚752370757662208 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 407307840 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10089194520064 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 184263868 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 110407955 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53429812 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53429183 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7885063769992092e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 _runtime‚10434.777032351 _step‚723 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 407307840 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10088284535296 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 407281468 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 163837138 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223104 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚8799 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚631 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 163837769 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚717693613623072 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 223043970 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚742282473126912 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚818911406507008 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚717693596583712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚742282491129856 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚629 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚727744164396277 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283008 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10050567812565 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 223017600 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 184263868 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚752371685649920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚752370757662208 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 407307840 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10089194520064 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 184263868 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 110407955 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53429812 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53429183 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7885063769992092e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 _runtime‚10434.777032351 _step‚723 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 407307840 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10088284535296 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 1255358771ôÒš—Ð2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 407281468 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 163837138 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223104 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚756425058832384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 _runtime‚10494.731276304 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚756425058832384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53713718 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 trainer/global_step‚8849 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 224237671 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚746281808332288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚746281808332288 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 409515725 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 110999142 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53714367 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10143250500096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 110999140 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚833923679408128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚746281826335232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10106235769399 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚731671711525127 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10143250500096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚649 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚731670786510763 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 224264015 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 409489379 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10590956466688 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _step‚726 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚833923422491648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 185251708 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 224237671 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10105327794395 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚756425986806784 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 224264015 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚756425986806784 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10144160471552 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10590956466688 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _step‚726 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚833923422491648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 185251708 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 224237671 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10105327794395 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚756425986806784 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 224264015 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚756425986806784 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10144160471552 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚756425058832384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 _runtime‚10494.731276304 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚756425058832384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53713718 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 trainer/global_step‚8849 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 224237671 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚746281808332288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚746281808332288 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 409515725 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 110999142 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53714367 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10143250500096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 110999140 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚833923679408128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚746281826335232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346ÖWÙ¥g4ð%/ _wandbruntime‚10495  _runtime‚10495¬—€ñ%‹N ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚746363445421056 % _timeù(¾‡‘%stamp‚1.7885064380580854e+09 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10107353042885 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 185271870 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53719506 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10145281945088 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 224288872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 224262502 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚731751846393157 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 111011205 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚746363445421056 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 409534370 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10106445067673 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 409560742 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _step‚727 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 224262502 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 164730711 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10592117161472 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10144371960320 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 185271868 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚834014808890880 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚823422691729408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 164731366 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚746363427418112 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚10495.836087601 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10144371960320 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10592350124544 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 409560742 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53720159 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚653 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚834015065794048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 185271868 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10145281945088Ý”a 'ò%N JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 111011205 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚746363445421056 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 409534370 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10106445067673 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 409560742 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _step‚727 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 224262502 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 164730711 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10592117161472 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10144371960320 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 185271868 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚834014808890880 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚823422691729408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 164731366 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚746363427418112 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚10495.836087601 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10144371960320 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10592350124544 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 409560742 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53720159 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚653 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚834015065794048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 185271868 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10145281945088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚746363445421056 % _timestamp‚1.7885064380580854e+09 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10107353042885 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 185271870 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53719506 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10145281945088 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 224288872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 224262502 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚731751846393157ף˗â4ó%/ _wandbruntime‚10495  _runtime‚10495µSôÍWô%Ñ trainer/global_step‚8849 $ _timestamp‚1.788506438060068e+09 _step‚728 0loss/FRt1/assignment_loss‚2.2672989472746847 1loss/SRqqt3/assignment_loss‚0.866120522756147 0loss/FBt2/assignment_loss‚0.2821943734656648 /loss/SRqqt4/detection_loss‚2.58358122587204 2loss/SRqqt4/assignment_loss‚1.0213631229770543 1loss/FBt3/assignment_loss‚0.24104560867328054 .loss/FRt2/detection_loss‚2.321119977068156 0loss/SRqqt2/detection_loss‚2.454637007713318 /loss/FBt4/detection_loss‚2.4541160901387533 0loss/FRt4/assignment_loss‚3.0994831181707836 1loss/FBt1/assignment_loss‚0.15597940496622287 /loss/FBt1/detection_loss‚0.5657774928684641 .loss/FRt4/detection_loss‚3.498311289719173 .loss/FBt2/detection_loss‚2.306979940487788 .loss/FBt3/detection_loss‚2.712645179950274 epoch‚7 _runtime‚10495.837089606 /loss/FRt2/assignment_loss‚3.406182715296745 2loss/SRqqt1/assignment_loss‚1.6746211685240269 .loss/FRt1/detection_loss‚1.815087971954199 /loss/FRt3/detection_loss‚1.9285871645063162 1loss/SRqqt3/detection_loss‚1.0543141259442803 0loss/FRt3/assignment_loss‚1.5621034300327301 2loss/SRqqt2/assignment_loss‚1.8921183919906617 1loss/SRqqt1/detection_loss‚2.0820376177628837 0loss/FBt4/assignment_loss‚0.4000583930045802 %loss/total_loss‚42.64576428111823ÔWÅt\õ%Ö 0loss/FBt4/assignment_loss‚0.4000583930045802 %loss/total_loss‚42.64576428111823 trainer/global_step‚8849 $ _timestamp‚1.788506438060068e+09 _step‚728 0loss/FRt1/assignment_loss‚2.2672989472746847 1loss/SRqqt3/assignment_loss‚0.866120522756147 0loss/FBt2/assignment_loss‚0.2821943734656648 /loss/SRqqt4/detection_loss‚2.58358122587204 2loss/SRqqt4/assignment_loss‚1.0213631229770543 1loss/FBt3/assignment_loss‚0.24104560867328054 .loss/FRt2/detection_loss‚2.321119977068156 0loss/SRqqt2/detection_loss‚2.454637007713318 /loss/FBt4/detection_loss‚2.4541160901387533 0loss/FRt4/assignment_loss‚3.0994831181707836 1loss/FBt1/assignment_loss‚0.15597940496622287 /loss/FBt1/detection_loss‚0.5657774928684641 .loss/FRt4/detection_loss‚3.498311289719173 .loss/FBt2/detection_loss‚2.306979940487788 .loss/FBt3/detection_loss‚2.712645179950274 epoch‚7 _runtime‚10495.837089606 /loss/FRt2/assignment_loss‚3.406182715296745 2loss/SRqqt1/assignment_loss‚1.6746211685240269 .loss/FRt1/detection_loss‚1.815087971954199 /loss/FRt3/detection_loss‚1.9285871645063162 1loss/SRqqt3/detection_loss‚1.0543141259442803 0loss/FRt3/assignment_loss‚1.5621034300327301 2loss/SRqqt2/assignment_loss‚1.8921183919906617 1loss/SRqqt1/detection_loss‚2.0820376177628837Ø«>·ö“ö%‚(:ˆ ÊâéÔäò¶m gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚156.911' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.22775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?…]™j÷%‚(:ß ÊâéÔÑÍþn disk.rbd2.in‚0 disk.rbd2.out‚ 3.033203125 disk.sda1.out‚0 disk.sdb1.in‚0 disk.md0p1.in‚ 12.296875 disk.sdb2.in‚3.78125! disk.sdb.out‚1169.89599609375 proc.cpu.threads‚17$ disk./.usageGB‚2711.150810241699 disk.md0.in‚ 12.296875! disk.sda.out‚1169.89599609375 disk.rbd0.in‚0! disk.dm-0.out‚155179.62890625$ memory_percent‚92.02008247375488 cpu‚25.202505206505965 disk.nvme1n1.in‚ 744.57421875# disk.nvme1n1.out‚43502.12890625 disk.sdb.in‚3.78125 disk.nvme0n1.out‚111677.5" disk.sda2.out‚1169.89599609375) disk./.usagePercent‚75.80491135912482 disk.sda1.in‚0 disk.sdb1.out‚0 disk.sda2.in‚8.515625 disk.nvme0n1.in‚ 939.6640625 network.sent‚ 2.3220066e+07 disk.md0.out‚ 1125.23046875" disk.sdb2.out‚1169.89599609375 disk.rbd0.out‚ 2.74609375 disk.dm-0.in‚ 1684.23828125 proc.memory.rssMB‚3013.625 network.recv‚ 3.386702e+06 disk.sda.in‚8.515625 disk.md0p1.out‚ 1125.23046875( proc.memory.availableMB‚ 1307.4296875* proc.memory.percent‚18.393707275390625å “ø%‚(:ˆ ÙâéÔìòŠm gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.727' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.18175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Â'Q aù%‚(:Ö ÙâéÔ§Ž¿m! disk.nvme1n1.out‚ 43602.578125( disk./.usagePercent‚75.8049129590328! disk.sda2.out‚1171.8330078125 disk.md0.in‚ 12.296875 disk.md0.out‚ 1127.09765625 disk.sda1.in‚0 network.sent‚ 2.331408e+07$ disk./.usageGB‚2711.150867462158! disk.sdb2.out‚1171.8330078125 disk.sda.out‚1171.8330078125 cpu‚25.202248207058247 network.recv‚ 3.391248e+06 disk.rbd2.in‚0 disk.sda.in‚8.515625 disk.sda1.out‚0 disk.md0p1.out‚ 1127.09765625$ memory_percent‚92.01853275299072) proc.memory.availableMB‚ 1307.68359375* proc.memory.percent‚18.393707275390625 disk.sda2.in‚8.515625 disk.nvme0n1.in‚ 939.6640625 disk.nvme1n1.in‚ 744.57421875 disk.sdb1.in‚0 proc.cpu.threads‚17 disk.dm-0.out‚ 155280.078125 disk.sdb.in‚3.78125 disk.sdb.out‚1171.8330078125 disk.rbd2.out‚ 3.033203125 disk.dm-0.in‚ 1684.23828125 disk.nvme0n1.out‚111677.5 disk.sdb2.in‚3.78125 disk.rbd0.in‚0 disk.md0p1.in‚ 12.296875 disk.rbd0.out‚ 2.74609375 disk.sdb1.out‚0 proc.memory.rssMB‚3013.625„u‚“ú%‚(:ˆ èâéÔ´åám gpu.0.gpu‚33.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚125.603' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.40075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øÝQmû%‚(:â èâéÔÅÂûm% disk./.usageGB‚2710.6198921203613 disk.md0p1.in‚ 12.296875 disk.nvme0n1.in‚ 939.6640625" disk.sdb2.out‚1173.33154296875 proc.memory.rssMB‚3013.625 cpu‚25.20198215213998! disk.dm-0.out‚155499.16796875 network.sent‚ 2.3320082e+07 disk.sda1.in‚0 disk.md0p1.out‚ 1128.54296875! disk.sdb.out‚1173.33154296875$ memory_percent‚92.01712608337402) disk./.usagePercent‚75.79006666624572 disk.sdb1.in‚0 disk.dm-0.in‚ 1684.23828125 disk.sdb2.in‚3.78125 disk.sda.in‚8.515625! disk.sda.out‚1173.33154296875 disk.rbd0.in‚0 disk.rbd0.out‚ 2.74609375 disk.rbd2.in‚0( proc.memory.availableMB‚ 1307.9140625* proc.memory.percent‚18.393707275390625 disk.sdb.in‚3.78125" disk.sda2.out‚1173.33154296875 disk.nvme1n1.in‚ 744.57421875 disk.md0.in‚ 12.296875 disk.nvme0n1.out‚ 111792.1875# disk.nvme1n1.out‚43706.98046875 disk.rbd2.out‚ 3.033203125 proc.cpu.threads‚17 disk.md0.out‚ 1128.54296875 network.recv‚ 3.395676e+06 disk.sda1.out‚0 disk.sdb1.out‚0 disk.sda2.in‚8.515625…¤Ô“ü%‚(:ˆ ÷âéÔ‚ŠŽm gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.305' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.07625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æƒäÀiý%‚(:Þ ÷âéÔ‡ˆïm disk.rbd2.in‚0' proc.memory.availableMB‚ 1307.921875 proc.memory.rssMB‚3013.625 cpu‚25.20174331198631 network.sent‚ 2.3328127e+07 disk.sdb1.in‚0 disk.md0.in‚ 12.296875 disk.sdb2.in‚3.78125) disk./.usagePercent‚75.79005920000846 disk.sdb1.out‚0 disk.sdb.in‚3.78125" disk.sda2.out‚1174.88330078125" disk.sdb2.out‚1174.88330078125 proc.cpu.threads‚17 disk.md0p1.out‚ 1130.03515625 disk.dm-0.out‚155547.9609375 disk.nvme0n1.in‚ 939.6640625 disk.md0p1.in‚ 12.296875 disk.nvme0n1.out‚ 111792.1875 disk.sda.in‚8.515625 disk.rbd0.in‚0! disk.sda.out‚1174.88330078125* proc.memory.percent‚18.393707275390625 disk.rbd0.out‚ 2.74609375 disk.rbd2.out‚ 3.033203125 network.recv‚ 3.399807e+06 disk.sda1.in‚0! disk.sdb.out‚1174.88330078125 disk.sda2.in‚8.515625 disk.nvme1n1.in‚ 744.57421875 disk.md0.out‚ 1130.03515625# memory_percent‚92.0170783996582% disk./.usageGB‚2710.6196250915527 disk.sda1.out‚0 disk.dm-0.in‚ 1684.23828125" disk.nvme1n1.out‚ 43755.7734375Ä°Éj4þ%/ _wandbruntime‚10554  _runtime‚10554Oe…Çÿ%Á & lr-AdamW/pg1‚0.0002079588406281598 & lr-AdamW/pg2‚0.0002079588406281598 trainer/global_step‚8899 $ _timestamp‚1.788506496983329e+09 _runtime‚10554.760735306 _step‚729—ð…®Ì€&Æ _runtime‚10554.760735306 _step‚729 & lr-AdamW/pg1‚0.0002079588406281598 & lr-AdamW/pg2‚0.0002079588406281598 trainer/global_step‚8899 $ _timestamp‚1.788506496983329e+09ÙÊN¨4&/ _wandbruntime‚10554  _runtime‚10554@ÛÙì ‚&€P YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚735678468507435 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10161205049503 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 165606604 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚838493258414592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 225508917 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚760563028523008 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10199337925120 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10199337925120 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚735677543493071 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚750362780626432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 225482573 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚750362762623488 A8DeviceStatsMonitor.on_train_batch_start/segmxɹ¡úent.all.peak‚1185 _step‚730 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 111602392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 165605984 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚760562100548608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 186259710 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚725516338443568 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚760563028523008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 111602390 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚827843751247360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 225482573 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 186259710 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10649507167232 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10200247896576 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚725516355482928 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚760562100548608 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚10554.762317493 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 411742281 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚750362780626432 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570aŒA (ƒ&…P N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚760563028523008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 111602390 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚827843751247360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 225482573 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 186259710 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10649507167232 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10200247896576 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚725516355482928 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚760562100548608 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚10554.762317493 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 411742281 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚750362780626432 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚735678468507435 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10161205049503 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 165606604 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚838493258414592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 225508917 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚760563028523008 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10199337925120 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10199337925120 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚735677543493071 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚750362780626432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 225482573 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚750362762623488 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 _step‚730 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 111602392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 165605984 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚760562100548608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 186259710 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚725516338443568 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640ÚÅH´4„&/ _wandbruntime‚10555  _runtime‚10555¡<Ÿ›'…&ŠN L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 411787272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 225507404 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _step‚731 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 411813644 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 411787272 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 186279868 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 225533774 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚838584665265152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚827933976951808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10650688313344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 111614455 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 _runtime‚10555.958184998 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 165623836 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚760645769082368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚750444399712256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 225533774 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚725595356038112 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10162322322781 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 trainer/global_step‚8899 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚760645769082368 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚750444381709312 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 225507404 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 165624440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54009381 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚735757678360893 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 186279870 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10200459385344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 411813644 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 186279868 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚750444399712256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54009983 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚725595373077472 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 111614457 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10201369370112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚602 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10201369370112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚838584911682560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 225507404 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 165624440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54009381 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚735757678360893 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 186279870 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10200459385344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 411813644 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 186279868 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚750444399712256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54009983 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚725595373077472 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 111614457 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10201369370112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚602 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10201369370112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚838584911682560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/allocatioÙ¡£—n.all.freed‚ 411787272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 225507404 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _step‚731 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 411813644 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 411787272 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 186279868 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 225533774 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚838584665265152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚827933976951808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10650688313344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 111614455 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 _runtime‚10555.958184998 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 165623836 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚760645769082368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚750444399712256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 225533774 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚725595356038112 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10162322322781 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 trainer/global_step‚8899 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚760645769082368 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚750444381709312 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0Û1Z†‘4‡&/ _wandbruntime‚10555  _runtime‚105555}i¸`ˆ&Ú 0loss/FRt3/assignment_loss‚1.3141998309513618 0loss/FBt2/assignment_loss‚0.4182828602097288 0loss/SRqqt4/detection_loss‚2.697854422032833 % _timestamp‚1.7885064981823514e+09 2loss/SRqqt4/assignment_loss‚1.1481857023512323 .loss/FRt2/detection_loss‚2.054011211148463 2loss/SRqqt3/assignment_loss‚1.0354392026793682 /loss/FRt1/detection_loss‚1.8086139492519404 /loss/FRt3/detection_loss‚1.5034593389979725 /loss/FBt1/detection_loss‚0.8001191297552321 trainer/global_step‚8899 2loss/SRqqt1/assignment_loss‚1.5519056036495245 /loss/FBt4/detection_loss‚2.5617648720741273 epoch‚7 0loss/FRt1/assignment_loss‚1.7717353290035611 2loss/SRqqt2/assignment_loss‚1.5516562913270557 /loss/FRt4/detection_loss‚3.1287416470050813 /loss/FBt2/detection_loss‚2.5348580121994018 /loss/FBt3/detection_loss‚2.1722247666782803 0loss/FRt4/assignment_loss‚2.7047759342193602 1loss/FBt3/assignment_loss‚0.13335942173580406 0loss/SRqqt2/detection_loss‚2.287548138814814 &loss/total_loss‚39.635780365083846 _step‚732 1loss/FBt1/assignment_loss‚0.09532236540804981 1loss/SRqqt1/detection_loss‚1.9336193822897398 1loss/SRqqt3/detection_loss‚0.8631063854368795 /loss/FRt2/assignment_loss‚3.289100435872873 1loss/FBt4/assignment_loss‚0.27589613199115776 _runtime‚10555.959643594sòö¯e‰&ß /loss/FRt2/assignment_loss‚3.289100435872873 1loss/FBt4/assignment_loss‚0.27589613199115776 _runtime‚10555.959643594 0loss/FRt3/assignment_loss‚1.3141998309513618 0loss/FBt2/assignment_loss‚0.4182828602097288 0loss/SRqqt4/detection_loss‚2.697854422032833 % _timestamp‚1.7885064981823514e+09 2loss/SRqqt4/assignment_loss‚1.1481857023512323 .loss/FRt2/detection_loss‚2.054011211148463 2loss/SRqqt3/assignment_loss‚1.0354392026793682 /loss/FRt1/detection_loss‚1.8086139492519404 /loss/FRt3/detection_loss‚1.5034593389979725 /loss/FBt1/detection_loss‚0.8001191297552321 trainer/global_step‚8899 2loss/SRqqt1/assignment_loss‚1.5519056036495245 /loss/FBt4/detection_loss‚2.5617648720741273 epoch‚7 0loss/FRt1/assignment_loss‚1.7717353290035611 2loss/SRqqt2/assignment_loss‚1.5516562913270557 /loss/FRt4/detection_loss‚3.1287416470050813 /loss/FBt2/detection_loss‚2.5348580121994018 /loss/FBt3/detection_loss‚2.1722247666782803 0loss/FRt4/assignment_loss‚2.7047759342193602 1loss/FBt3/assignment_loss‚0.13335942173580406 0loss/SRqqt2/detection_loss‚2.287548138814814 &loss/total_loss‚39.635780365083846 _step‚732 1loss/FBt1/assignment_loss‚0.09532236540804981 1loss/SRqqt1/detection_loss‚1.9336193822897398 1loss/SRqqt3/detection_loss‚0.8631063854368795Üí0BŽŠ&‚(:ƒ †ãéÔª¥ûl gpu.0.gpu‚5.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚161.7' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Psöi‹&‚(:Þ †ãéÔäãn disk.dm-0.in‚ 1684.23828125 proc.memory.rssMB‚3013.625$ disk.nvme0n1.out‚112021.56640625 disk.md0.out‚ 1131.91015625 disk.rbd2.out‚ 3.033203125 disk.sdb2.out‚1176.826171875 disk.sda.out‚1176.826171875 disk.sda1.out‚0 disk.rbd0.out‚ 2.74609375) proc.memory.availableMB‚ 1307.61328125 network.recv‚ 3.404389e+06) disk./.usagePercent‚75.79079110457988 disk.md0.in‚ 12.296875 disk.md0p1.in‚ 12.296875 proc.cpu.threads‚17 disk.sdb1.out‚0! disk.dm-0.out‚155778.12109375 disk.sdb.out‚1176.826171875* proc.memory.percent‚18.393707275390625" disk.nvme1n1.out‚ 43756.5546875 disk.sdb2.in‚3.78125% disk./.usageGB‚2710.6458015441895 disk.rbd0.in‚0 disk.sda2.out‚1176.826171875# memory_percent‚92.0189619064331 cpu‚25.201502816403995 disk.nvme1n1.in‚ 744.57421875 disk.sda.in‚8.515625 disk.sda1.in‚0 disk.md0p1.out‚ 1131.91015625 disk.sda2.in‚8.515625 disk.sdb.in‚3.78125 network.sent‚ 2.3345045e+07 disk.nvme0n1.in‚ 939.6640625 disk.rbd2.in‚0 disk.sdb1.in‚0XñŸU“Œ&‚(:ˆ •ãéÔÝÁŠm gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚152.899' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.22475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ’»e&‚(:Ú •ãéÔÿ˜ñm disk.rbd0.out‚ 2.74609375% disk./.usageGB‚2710.6477661132812 disk.nvme1n1.in‚ 744.57421875 disk.rbd2.out‚ 3.033203125 disk.sda1.out‚0! disk.dm-0.out‚155969.80859375 disk.sda2.in‚8.515625 proc.memory.rssMB‚3013.625 disk.sda.in‚8.515625 network.recv‚ 3.40904e+06 disk.md0.out‚ 1133.62109375 disk.sdb1.in‚0 disk.md0p1.in‚ 12.296875 disk.sdb.out‚1178.603515625$ disk.nvme0n1.out‚112213.01953125 disk.sda.out‚1178.603515625 disk.sdb1.out‚0 disk.sdb.in‚3.78125 proc.cpu.threads‚17 disk.sdb2.in‚3.78125) disk./.usagePercent‚75.79084603475393 disk.md0p1.out‚ 1133.62109375 disk.sda2.out‚1178.603515625' proc.memory.availableMB‚ 1307.578125 network.sent‚ 2.344127e+07 disk.sdb2.out‚1178.603515625 disk.sda1.in‚0 disk.dm-0.in‚ 1684.23828125 disk.nvme0n1.in‚ 939.6640625# memory_percent‚92.0191764831543 cpu‚25.201266605044076 disk.md0.in‚ 12.296875 disk.rbd2.in‚0 disk.rbd0.in‚0* proc.memory.percent‚18.393707275390625" disk.nvme1n1.out‚ 43756.7890625®W•á“Ž&‚(:ˆ ¤ãéÔÙŒ¡m gpu.0.gpu‚19.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚177.411' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.35275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡›m&‚(:â ¤ãéÔãÿ‘n" disk.sda2.out‚1180.17333984375 disk.nvme0n1.in‚ 939.6640625 disk.nvme1n1.in‚ 744.57421875" disk.sdb2.out‚1180.17333984375 disk.rbd0.in‚0 disk.sdb.in‚3.78125* proc.memory.percent‚18.393707275390625 disk.dm-0.in‚ 1684.23828125 disk.md0.out‚ 1135.1328125 disk.sda1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚75.79433938050198$ disk./.usageGB‚2710.772705078125 disk.rbd0.out‚ 2.74609375$ memory_percent‚92.01915264129639 disk.sdb2.in‚3.78125 disk.sdb1.in‚0) proc.memory.availableMB‚ 1307.58203125! disk.dm-0.out‚156077.17578125 disk.md0.in‚ 12.296875 disk.sda.in‚8.515625! disk.sda.out‚1180.17333984375 disk.md0p1.in‚ 12.296875 disk.md0p1.out‚ 1135.1328125# disk.nvme0n1.out‚112319.8984375 disk.rbd2.in‚0 disk.rbd2.out‚ 3.033203125 disk.sdb1.out‚0! disk.sdb.out‚1180.17333984375# disk.nvme1n1.out‚43757.27734375 network.sent‚ 2.344741e+07 network.recv‚ 3.413619e+06 disk.sda2.in‚8.515625 disk.sda1.in‚0 proc.memory.rssMB‚3013.625 cpu‚25.20102808949326"±ä¡‘&‚(:† ³ãéÔãôÄm gpu.0.gpu‚33.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚121.876' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.469 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Kåç\‘&‚(:Ñ ³ãéÔÞÿm disk.sdb1.out‚0! disk.dm-0.out‚156232.25390625 disk.nvme0n1.in‚ 939.6640625 network.recv‚ 3.4174e+06 disk.nvme1n1.in‚ 744.57421875 disk.sdb2.in‚3.78125 disk.dm-0.in‚ 1684.23828125! disk.sdb.out‚1182.11083984375 disk.sda1.out‚0" disk.sda2.out‚1182.11083984375$ disk.nvme0n1.out‚112474.84765625$ memory_percent‚92.01900959014893% disk./.usageGB‚2710.7452812194824" disk.sdb2.out‚1182.11083984375 cpu‚25.200792121404408 disk.nvme1n1.out‚ 43757.40625 disk.md0.out‚1137 disk.md0p1.in‚ 12.296875) proc.memory.availableMB‚ 1307.60546875 disk.rbd0.out‚ 2.74609375 proc.memory.rssMB‚3013.625 proc.cpu.threads‚17) disk./.usagePercent‚75.79357259793656 disk.rbd2.out‚ 3.033203125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sda2.in‚8.515625! disk.sda.out‚1182.11083984375 network.sent‚ 2.3453217e+07 disk.rbd2.in‚0 disk.sda.in‚8.515625 disk.md0p1.out‚1137 disk.sdb.in‚3.78125* proc.memory.percent‚18.393707275390625 disk.md0.in‚ 12.296875 ¨¸k4’&/ _wandbruntime‚10612  _runtime‚10612‹Kˆ<Ê“&Ä % _timestamp‚1.7885065548427835e+09 _runtime‚10612.620018253 _step‚733 ' lr-AdamW/pg1‚0.00020194280523071214 ' lr-AdamW/pg2‚0.00020194280523071214 trainer/global_step‚8949ŸV:õÏ”&É _runtime‚10612.620018253 _step‚733 ' lr-AdamW/pg1‚0.00020194280523071214 ' lr-AdamW/pg2‚0.00020194280523071214 trainer/global_step‚8949 % _timestamp‚1.7885065548427835e+09Ý6x“p4•&/ _wandbruntime‚10612  _runtime‚10612> ü(–&P D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10255425350144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚832355060409856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚729467235210128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10256335321600 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 414021529 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 166499779 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10708210100736 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚764700070239232 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚843063270510592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚739685225489743 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 166499153 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚764699142264832 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚624 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _step‚734 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 187267710 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 226753819 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 413995183 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚754443734917632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 226727475 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚739684300475379 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10217082304611 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚764700070239232 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10256335321600 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚626 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚754443716914688 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 112205642 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 187267710 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0Z½$"—&†P _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10217990279615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚754443716914688 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚843063015691264 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚832355036469760 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10707979221504 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _runtime‚10612.621133328 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 112205640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54293513 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 187267708 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚739684300475379 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10217082304611 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚764700070239232 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10256335321600 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚626 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚754443716914688 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 112205642 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 187267710 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10255425350144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚832355060409856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚729467235210128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10256335321600 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 414021529 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 166499779 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10708210100736 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚764700070239232 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚843063270510592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚739685225489743 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 166499153 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚764699142264832 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚624 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _step‚734 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 187267710 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 226753819 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 413995183 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 GADeviceStatsMonitor.on_train_batch_start/active.large_¨ùšèpool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚754443734917632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 226727475 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 166517028 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54299965 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 187287868 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 414066546 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10219107553101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚832445286114304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚739764435343201 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 226778676 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚8949 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚754525354003456 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚754525354003456 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10257456795136 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54299323 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚843154662778368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚739765360357773 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚764782810798592 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 187287870 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 187287870 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 226752306 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 226778676 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚843154412166656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 _step‚735 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 112217707 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10709149992448 % _timestamp‚1.7885065557804453e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 187287868 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚754525336000512 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2¢¦Èï'š&N B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54299323 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚843154662778368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚739765360357773 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚764782810798592 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 187287870 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 187287870 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 226752306 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 226778676 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚843154412166656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 _step‚735 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 112217707 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10709149992448 % _timestamp‚1.7885065557804453e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 187287868 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚754525336000512 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 112217705 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚832445262174208 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10709376664064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10257456795136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10218199577889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 226752306 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 166517028 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54299965 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 187287868 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 414066546 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10219107553101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚832445286114304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚739764435343201 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 226778676 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚8949 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚754525354003456 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚754525354003456 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10257456795136 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768ßJk®4›&/ _wandbruntime‚10613  _runtime‚10613^20aœ&Û /loss/FRt4/assignment_loss‚2.485123666624228 1loss/FBt4/assignment_loss‚0.26116165455304824 0loss/FRt1/assignment_loss‚2.4449379218591227 2loss/SRqqt3/assignment_loss‚0.9638515651070824 0loss/FBt3/assignment_loss‚0.3500307349386276 /loss/FBt1/detection_loss‚0.5438520217513931 /loss/FBt4/detection_loss‚2.2043680250644684 2loss/SRqqt1/assignment_loss‚1.4599555689427588 1loss/SRqqt2/assignment_loss‚1.392214741682013 1loss/FBt2/assignment_loss‚0.10963467820632407 /loss/FRt2/detection_loss‚2.8529006212185593 /loss/FBt2/detection_loss‚2.4485075374444327 _step‚736 0loss/FRt2/assignment_loss‚2.6554673390510755 2loss/SRqqt4/assignment_loss‚1.5387948036193848 .loss/FRt3/detection_loss‚1.493309073150158 1loss/SRqqt4/detection_loss‚2.3245680212974547 trainer/global_step‚8949 0loss/FRt3/assignment_loss‚1.5633345702663064 /loss/FRt4/detection_loss‚2.8972424690922103 /loss/FBt3/detection_loss‚2.5010538175702095 /loss/FRt1/detection_loss‚2.2309780316096983 1loss/FBt1/assignment_loss‚0.19932951842155586 1loss/SRqqt2/detection_loss‚2.0743756026029585 % _timestamp‚1.7885065557827988e+09 _runtime‚10613.559794208 1loss/SRqqt1/detection_loss‚2.5516920304960675 1loss/SRqqt3/detection_loss‚0.8549466785509139 &loss/total_loss‚40.401630693120055 epoch‚7¶é½f&à 1loss/FBt1/assignment_loss‚0.19932951842155586 1loss/SRqqt2/detection_loss‚2.0743756026029585 % _timestamp‚1.7885065557827988e+09 _runtime‚10613.559794208 1loss/SRqqt1/detection_loss‚2.5516920304960675 1loss/SRqqt3/detection_loss‚0.8549466785509139 &loss/total_loss‚40.401630693120055 epoch‚7 /loss/FRt4/assignment_loss‚2.485123666624228 1loss/FBt4/assignment_loss‚0.26116165455304824 0loss/FRt1/assignment_loss‚2.4449379218591227 2loss/SRqqt3/assignment_loss‚0.9638515651070824 0loss/FBt3/assignment_loss‚0.3500307349386276 /loss/FBt1/detection_loss‚0.5438520217513931 /loss/FBt4/detection_loss‚2.2043680250644684 2loss/SRqqt1/assignment_loss‚1.4599555689427588 1loss/SRqqt2/assignment_loss‚1.392214741682013 1loss/FBt2/assignment_loss‚0.10963467820632407 /loss/FRt2/detection_loss‚2.8529006212185593 /loss/FBt2/detection_loss‚2.4485075374444327 _step‚736 0loss/FRt2/assignment_loss‚2.6554673390510755 2loss/SRqqt4/assignment_loss‚1.5387948036193848 .loss/FRt3/detection_loss‚1.493309073150158 1loss/SRqqt4/detection_loss‚2.3245680212974547 trainer/global_step‚8949 0loss/FRt3/assignment_loss‚1.5633345702663064 /loss/FRt4/detection_loss‚2.8972424690922103 /loss/FBt3/detection_loss‚2.5010538175702095 /loss/FRt1/detection_loss‚2.2309780316096983à–½»*’ž&‚(:‡ ÂãéÔÅÙÏk gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.058' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.7645 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿ñIî`Ÿ&‚(:Õ ÂãéÔŸëÞm disk.md0p1.out‚ 1138.59375 disk.dm-0.in‚ 1684.23828125! disk.sda2.out‚1183.7822265625 disk.nvme1n1.in‚ 744.57421875 disk.md0.in‚ 12.296875 disk.rbd2.in‚0 disk.md0p1.in‚ 12.296875 disk.sdb.out‚1183.7822265625 disk.sdb2.in‚3.78125 disk.rbd0.out‚ 2.74609375" disk.nvme1n1.out‚ 43757.9453125' proc.memory.availableMB‚ 1307.078125 proc.memory.rssMB‚3013.625* proc.memory.percent‚18.393707275390625 disk.sdb1.in‚0 cpu‚25.200533914442577 disk.sdb1.out‚0 disk.sda2.in‚8.515625 disk.nvme0n1.in‚ 939.6640625 disk.dm-0.out‚156474.1484375 disk.md0.out‚ 1138.59375 network.recv‚ 3.422392e+06( disk./.usagePercent‚75.7752378657859 disk.sda.in‚8.515625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sdb.in‚3.78125" disk.nvme0n1.out‚ 112716.203125 network.sent‚ 2.3457393e+07$ disk./.usageGB‚2710.089542388916 disk.rbd2.out‚ 3.033203125# memory_percent‚92.0222282409668 proc.cpu.threads‚17! disk.sdb2.out‚1183.7822265625 disk.sda.out‚1183.7822265625dùÎ “ &‚(:ˆ ÑãéÔŒÆÍm gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“9Ìël¡&‚(:á ÑãéÔüæˆn disk.rbd2.in‚0 disk.sdb2.in‚3.78125 disk.sdb1.in‚0# disk.nvme0n1.out‚112802.7890625 disk.rbd2.out‚ 3.033203125! disk.sdb2.out‚1185.2255859375 disk.dm-0.in‚ 1684.23828125 disk.sda2.in‚8.515625 disk.nvme0n1.in‚ 939.6640625! disk.dm-0.out‚156560.88671875 network.sent‚ 2.3565734e+07) disk./.usagePercent‚75.77523903905174 disk.sda.out‚1185.2255859375 disk.sda1.in‚0 disk.sdb.out‚1185.2255859375! disk.sda2.out‚1185.2255859375 cpu‚25.200297050362234 disk.md0p1.in‚ 12.296875 disk.md0p1.out‚ 1139.98828125 proc.memory.rssMB‚3013.625* proc.memory.percent‚18.393707275390625 proc.cpu.threads‚17 disk.rbd0.out‚ 2.74609375 disk.sdb1.out‚0 disk.sda.in‚8.515625 disk.md0.out‚ 1139.98828125$ memory_percent‚92.01922416687012( proc.memory.availableMB‚ 1307.5703125$ disk./.usageGB‚2710.089584350586 disk.sda1.out‚0 disk.sdb.in‚3.78125 disk.nvme1n1.in‚ 744.57421875 disk.md0.in‚ 12.296875 network.recv‚ 3.426123e+06 disk.rbd0.in‚0# disk.nvme1n1.out‚43758.09765625tîš}¢&‚(:… àãéÔ¡•Îk gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚161.864' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SNoú`£&‚(:Õ àãéÔûîŒn disk.sdb1.in‚0 disk.rbd0.out‚ 2.74609375# disk.nvme0n1.out‚112976.6953125 disk.nvme1n1.in‚ 744.57421875# disk.nvme1n1.out‚43758.93359375 disk.md0.out‚ 1141.6484375$ disk./.usageGB‚2710.089530944824 disk.sdb2.in‚3.78125 disk.sda1.in‚0 disk.md0p1.in‚ 12.296875 disk.sdb.out‚ 1186.95703125 disk.md0.in‚ 12.296875 proc.memory.rssMB‚3013.625 disk.rbd2.in‚0 disk.sdb2.out‚ 1186.95703125 disk.dm-0.in‚ 1684.23828125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sdb.in‚3.78125* proc.memory.percent‚18.393707275390625 network.recv‚ 3.430476e+06( disk./.usagePercent‚75.7752375458043 disk.sda1.out‚0$ memory_percent‚92.02220439910889 disk.sda2.in‚8.515625 disk.sda2.out‚ 1186.95703125 network.sent‚ 2.3571778e+07 disk.sda.in‚8.515625 disk.sda.out‚ 1186.95703125 cpu‚25.2000389057571 disk.rbd2.out‚ 3.033203125 disk.md0p1.out‚ 1141.6484375! disk.dm-0.out‚156735.62890625 disk.nvme0n1.in‚ 939.6640625) proc.memory.availableMB‚ 1307.08203125ÎÖßü’¤&‚(:‡ ïãéÔü’¾m gpu.0.gpu‚13.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.562' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.6405 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~eƒm¥&‚(:â ïãéÔé—n disk.sdb1.in‚0" disk.nvme1n1.out‚ 43759.0859375( proc.memory.availableMB‚ 1307.5859375" disk.nvme0n1.out‚ 112977.765625 disk.md0.out‚ 1145.22265625 disk.rbd2.in‚0 disk.dm-0.out‚156736.8515625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sdb1.out‚0" disk.sda2.out‚1190.59912109375 disk.nvme1n1.in‚ 744.57421875 proc.memory.rssMB‚3013.625 disk.rbd0.in‚0! disk.sdb.out‚1190.59912109375 disk.nvme0n1.in‚ 939.6640625$ memory_percent‚92.01912879943848* proc.memory.percent‚18.393707275390625 network.recv‚ 3.434584e+06) disk./.usagePercent‚75.75772335312725 disk.rbd0.out‚ 2.74609375 disk.md0p1.out‚ 1145.22265625 disk.sda2.in‚8.515625 network.sent‚ 2.3577574e+07 disk.md0.in‚ 12.296875 disk.rbd2.out‚ 3.033203125 disk.md0p1.in‚ 12.296875 disk.sdb.in‚3.78125 cpu‚25.19980454379518 disk.sdb2.in‚3.78125" disk.sdb2.out‚1190.59912109375 disk.sda.in‚8.515625 disk.sda1.out‚0 disk.dm-0.in‚ 1684.23828125% disk./.usageGB‚2709.4631385803223! disk.sda.out‚1190.59912109375³Ó¦4¦&/ _wandbruntime‚10665  _runtime‚10665îôýʧ&Ä ' lr-AdamW/pg1‚0.00019599433972590733 ' lr-AdamW/pg2‚0.00019599433972590733 trainer/global_step‚8999 % _timestamp‚1.7885066079780562e+09 _runtime‚10665.755342869 _step‚737$Œn-Ϩ&É trainer/global_step‚8999 % _timestamp‚1.7885066079780562e+09 _runtime‚10665.755342869 _step‚737 ' lr-AdamW/pg1‚0.00019599433972590733 ' lr-AdamW/pg2‚0.00019599433972590733áT€Ïê4©&/ _wandbruntime‚10665  _runtime‚10665-:Uª&€P _step‚738 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10766615177216 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚768837111955456 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 167392251 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚768837111955456 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 416248085 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚758524689208832 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚847632716475904 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10766394783744 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚733418097897968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 227972377 TGDeviceS‡Âr™±!tatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 trainer/global_step‚8999 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 227998721 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54583970 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 112808890 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚836866345632256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚768836183981056 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚768836183981056 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 188275710 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 227998721 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 227972377 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 188275708 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 416274431 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚836866321692160 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚758524689208832 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚758524671205888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54583361 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10311512775168 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10311512775168 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 112808892 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10312422746624 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚743691982472051 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10273867534723 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 167392862 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚847632960809472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚758524671205888 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 188275710 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _runtime‚10665.756715825 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚733418114937328 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 188275708 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚609 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10312422746624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚836866345632256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚768836183981056 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚768836183981056 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 188275710 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 227998721 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 227972377 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 188275708 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 416274431 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚836866321692160 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚758524689208832 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚758524671205888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54583361 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10311512775168 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10311512775168 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 112808892 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10312422746624 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚743691982472051 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10273867534723 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 167392862 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚847632960809472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚758524671205888 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 188275710 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _runtime‚10665.756715825 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚733418114937328 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 188275708 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚609 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10312422746624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 112808890 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 188295870 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _runtime‚10666.698126668 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 416319448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 188295868 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10274984808209 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 228023578 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚758606308294656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚743771192325509 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚768919852514816 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 112820955 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚743772117340081 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚733497115492512 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 167410141 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 416319448 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 416293076 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 167410752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 228023578 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚768918924527104 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10767586745856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 trainer/global_step‚8999 _step‚739 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 188295870 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _runtime‚10666.698126668 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 416319448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 188295868 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10274984808209 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 228023578 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚758606308294656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚743771192325509 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_¸k¤¦6batch_end/reserved_bytes.small_pool.current‚ 1195376640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚768919852514816 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 112820955 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚743772117340081 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚733497115492512 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 167410141 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 416319448 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 416293076 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 167410752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 228023578 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚768918924527104 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10767586745856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 trainer/global_step‚8999 _step‚739 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ã¤pÍÙ4¯&/ _wandbruntime‚10666  _runtime‚10666øF%\°&Ö /loss/FRt2/detection_loss‚2.2761564016342164 2loss/SRqqt3/assignment_loss‚0.9663691402312986 1loss/FBt1/assignment_loss‚0.20958898187522115 $ _timestamp‚1.788506608922149e+09 _runtime‚10666.699223273 .loss/FBt3/detection_loss‚2.604369248662676 .loss/FBt4/detection_loss‚2.541113178838383 %loss/total_loss‚40.61045649157988 epoch‚7 /loss/FRt4/assignment_loss‚3.152952391367692 0loss/SRqqt2/detection_loss‚2.970264974236488 _step‚740 2loss/SRqqt1/assignment_loss‚0.8680904368973441 trainer/global_step‚8999 2loss/SRqqt2/assignment_loss‚0.9678690213710069 /loss/FRt3/detection_loss‚1.6235747423022986 /loss/FBt2/detection_loss‚2.7285274505615233 0loss/FRt1/assignment_loss‚2.0696240685604237 0loss/FRt2/assignment_loss‚2.9218653480211896 0loss/FRt3/assignment_loss‚1.4902000983556112 1loss/FBt2/assignment_loss‚0.10333638176980457 1loss/SRqqt3/detection_loss‚0.9540523591572824 2loss/SRqqt4/assignment_loss‚2.0288149222083716 0loss/FBt4/assignment_loss‚0.2462464427147676 .loss/FRt4/detection_loss‚2.976409985468938 0loss/SRqqt1/detection_loss‚2.288901953233613 1loss/SRqqt4/detection_loss‚2.4710751642351565 /loss/FBt1/detection_loss‚0.5934484717436135 1loss/FBt3/assignment_loss‚0.03372236931214242 /loss/FRt1/detection_loss‚1.5238829588208185SóWa±&Û 0loss/FRt1/assignment_loss‚2.0696240685604237 0loss/FRt2/assignment_loss‚2.9218653480211896 0loss/FRt3/assignment_loss‚1.4902000983556112 1loss/FBt2/assignment_loss‚0.10333638176980457 1loss/SRqqt3/detection_loss‚0.9540523591572824 2loss/SRqqt4/assignment_loss‚2.0288149222083716 0loss/FBt4/assignment_loss‚0.2462464427147676 .loss/FRt4/detection_loss‚2.976409985468938 0loss/SRqqt1/detection_loss‚2.288901953233613 1loss/SRqqt4/detection_loss‚2.4710751642351565 /loss/FBt1/detection_loss‚0.5934484717436135 1loss/FBt3/assignment_loss‚0.03372236931214242 /loss/FRt1/detection_loss‚1.5238829588208185 /loss/FRt2/detection_loss‚2.2761564016342164 2loss/SRqqt3/assignment_loss‚0.9663691402312986 1loss/FBt1/assignment_loss‚0.20958898187522115 $ _timestamp‚1.788506608922149e+09 _runtime‚10666.699223273 .loss/FBt3/detection_loss‚2.604369248662676 .loss/FBt4/detection_loss‚2.541113178838383 %loss/total_loss‚40.61045649157988 epoch‚7 /loss/FRt4/assignment_loss‚3.152952391367692 0loss/SRqqt2/detection_loss‚2.970264974236488 _step‚740 2loss/SRqqt1/assignment_loss‚0.8680904368973441 trainer/global_step‚8999 2loss/SRqqt2/assignment_loss‚0.9678690213710069 /loss/FRt3/detection_loss‚1.6235747423022986 /loss/FBt2/detection_loss‚2.7285274505615233ä!Ö;“²&‚(:ˆ þãéÔÜ×ßm gpu.0.gpu‚12.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚173.059' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.26475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸóØÏS³&‚(:È þãéÔƆ®n disk.md0p1.in‚ 12.296875 disk.sda2.out‚ 1192.03125# disk.nvme0n1.out‚113110.6015625 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.nvme0n1.in‚ 939.6640625 disk.nvme1n1.in‚ 744.57421875 disk.sda.out‚ 1192.03125 disk.sdb1.in‚0 disk.sdb.in‚3.78125 disk.sdb2.out‚ 1192.03125$ memory_percent‚92.02053546905518 cpu‚25.199570408730047$ disk./.usageGB‚2709.565845489502 disk.rbd0.in‚0 disk.sda2.in‚8.515625 disk.md0.in‚ 12.296875 disk.md0.out‚ 1146.59765625 disk.rbd2.in‚0 disk.sda1.in‚0* proc.memory.percent‚18.393707275390625 network.recv‚ 3.439284e+06 disk.rbd2.out‚ 3.048828125 disk.sdb2.in‚3.78125 disk.sda.in‚8.515625 proc.memory.rssMB‚3013.625 disk.rbd0.out‚ 2.7578125 disk.md0p1.out‚ 1146.59765625 disk.dm-0.out‚ 156875.34375" disk.nvme1n1.out‚ 43764.7421875 network.sent‚ 2.3581649e+07 disk.dm-0.in‚ 1684.23828125) proc.memory.availableMB‚ 1307.35546875( disk./.usagePercent‚75.7605950812941 disk.sdb.out‚ 1192.03125ٛ듴&‚(:ˆ äéÔäŽm gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚158.137' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.53425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"…djlµ&‚(:á äéÔ€ŒÞm proc.memory.rssMB‚3013.625) disk./.usagePercent‚75.76268818757646 disk.rbd0.in‚0 disk.md0p1.in‚ 12.296875! disk.sda.out‚1193.63623046875! disk.dm-0.out‚157030.62890625 disk.sdb.in‚3.78125 proc.cpu.threads‚17 disk.sdb1.in‚0# disk.nvme1n1.out‚43841.90234375# disk.nvme0n1.out‚113188.7265625 disk.md0.out‚ 1148.12890625 disk.sda1.out‚0 disk.sdb1.out‚0$ memory_percent‚92.02232360839844 cpu‚25.19931760128219 network.sent‚ 2.3690838e+07 disk.md0.in‚ 12.296875 disk.rbd2.out‚ 3.111328125 disk.sdb2.in‚3.78125 disk.md0p1.out‚ 1148.12890625 disk.sda2.in‚8.515625 disk.rbd2.in‚0* proc.memory.percent‚18.393707275390625 network.recv‚ 3.443902e+06% disk./.usageGB‚2709.6407051086426 disk.sda.in‚8.515625" disk.sda2.out‚1193.63623046875 disk.nvme1n1.in‚ 744.57421875% proc.memory.availableMB‚ 1307.0625 disk.nvme0n1.in‚ 939.6640625" disk.sdb2.out‚1193.63623046875 disk.sda1.in‚0 disk.rbd0.out‚ 2.7578125 disk.dm-0.in‚ 1684.23828125! disk.sdb.out‚1193.63623046875ÇÆÄ’¶&‚(:‡ œäéÔéôúl gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚161.254' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.3135 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Oܘ <·&‚(:± œäéÔš­ˆn disk.rbd2.out‚ 3.111328125 disk.sda1.out‚0* proc.memory.percent‚18.393707275390625 network.sent‚ 2.3696774e+07 disk.sdb.in‚3.78125 disk.sda2.out‚1195.5 disk.nvme0n1.in‚ 939.6640625 disk.md0p1.in‚ 12.296875 disk.dm-0.in‚ 1684.23828125 proc.memory.rssMB‚3013.625% disk./.usageGB‚2709.7103157043457 disk.sda.out‚1195.5$ memory_percent‚92.02234745025635 disk.sdb.out‚1195.5 disk.rbd0.in‚0 disk.rbd0.out‚ 2.8203125 disk.md0p1.out‚ 1149.92578125 disk.sdb1.in‚0 disk.md0.in‚ 12.296875 disk.md0.out‚ 1149.92578125 disk.rbd2.in‚0 disk.sdb2.in‚3.78125 disk.sda.in‚8.515625 disk.sda1.in‚0 disk.dm-0.out‚157067.5 network.recv‚ 3.448272e+06) disk./.usagePercent‚75.76463452896661 disk.sda2.in‚8.515625 disk.nvme1n1.in‚ 744.57421875 disk.sdb2.out‚1195.5) proc.memory.availableMB‚ 1307.05859375 cpu‚25.19908403783544 proc.cpu.threads‚17! disk.nvme0n1.out‚ 113194.09375 disk.nvme1n1.out‚ 43873.40625 disk.sdb1.out‚0 ·ú`4¸&/ _wandbruntime‚10722  _runtime‚10722”ocǹ&Á & lr-AdamW/pg1‚0.0001901146837650404 & lr-AdamW/pg2‚0.0001901146837650404 trainer/global_step‚9049 % _timestamp‚1.7885066649243612e+09 _runtime‚10722.70184983 _step‚741û=wÇ̺&Æ % _timestamp‚1.7885066649243612e+09 _runtime‚10722.70184983 _step‚741 & lr-AdamW/pg1‚0.0001901146837650404 & lr-AdamW/pg2‚0.0001901146837650404 trainer/global_step‚9049åÔ¥Èå4»&/ _wandbruntime‚10722  _runtime‚10722å“4(¼&P JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 168286624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 _step‚742 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚852202766738944 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10329744789831 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 189283710 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚747698739454359 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10367600200192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 229243623 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 189283708 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚762605643500032 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10367600200192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚737368994664528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 113412140 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚762605625497088 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 418527333 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10368510171648 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 265305088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚772974153671680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 229217279 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 % _timestamp‚1.7885066649262025e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚772974153671680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚646 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚9049 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 229217279 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 241364992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚772973225697280 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚737368977625168 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 113412142 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 168287270 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚841377606914560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚762605643500032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54875128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚644 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚10722.703775479 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 189283710 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10328836814827 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚852202501433856€˜Xš´ ½&†P XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10367600200192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚737368994664528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 113412140 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚762605625497088 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 418527333 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10368510171648 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 265305088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚772974153671680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 229217279 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 % _timestamp‚1.7885066649262025e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚772974153671680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚646 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚9049 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 229217279 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 241364992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚772973225697280 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚737368977625168 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 113412142 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 168287270 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚841377606914560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚762605643500032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54875128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚644 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚10722.703775479 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 189283710 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10328836814827 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚852202501433856 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 168286624 MFDeviceStatsMonitor.on_train_batch_start/in ÍcXactive_split.large_pool.peak‚44 _step‚742 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚852202766738944 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10329744789831 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 189283710 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚747698739454359 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10367600200192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 229243623 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 189283708 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚762605643500032æ'2Y4¾&/ _wandbruntime‚10723  _runtime‚10723.4"T'¿&ŠN K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚762687262585856 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 418572350 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚841467856559104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 229268480 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚747777949307817 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚773055966243328 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10368721660416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10826292624384 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10368721660416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 113424207 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 189303870 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚762687244582912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54880910 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 189303868 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54880277 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚841467832619008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _step‚743 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚762687244582912 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 229242110 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚747778874322389 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 418572350 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚10723.899133908 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10369631645184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 229242110 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚737448012259072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 168304482 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10329954088105 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 113424205 $ _timestamp‚1.788506666119831e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 189303870 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 189303868 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 418545978 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚633 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 418545978 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚852293898571776 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚737447995219712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10330862063317 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚852294149183488 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚773056894231040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 trainer/global_step‚9049êVó©'À&N SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 113424205 $ _timestamp‚1.788506666119831e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 189303870 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 189303868 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 418545978 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚633 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 418545978 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚852293898571776 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚737447995219712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10330862063317 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚852294149183488 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚773056894231040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 trainer/global_step‚9049 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚762687262585856 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 418572350 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚841467856559104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 229268480 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚747777949307817 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚773055966243328 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10368721660416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10826292624384 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10368721660416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 113424207 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 189303870 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚762687244582912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54880910 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 189303868 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54880277 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚841467832619008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _step‚743 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚762687244582912 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 229242110 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚747778874322389 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 418572350 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚10723.899133908 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10369631645184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 229242110 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚737448012259072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 168304482 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10329954088105 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096ç‡q—Ì4Á&/ _wandbruntime‚10723  _runtime‚10723,‘ÕÌ^Â&Ø 2loss/SRqqt4/assignment_loss‚1.4151269878176125 /loss/FRt2/detection_loss‚2.5764584037842173 0loss/SRqqt4/detection_loss‚2.793613302436742 .loss/FBt4/detection_loss‚2.443286540607611 trainer/global_step‚9049 2loss/SRqqt1/assignment_loss‚1.0743721105779211 1loss/FBt1/assignment_loss‚0.27740730934488234 /loss/FBt1/detection_loss‚0.7042060270468937 _runtime‚10723.900735215 1loss/FBt4/assignment_loss‚0.18760927295983265 /loss/FRt1/detection_loss‚1.6532682895955002 /loss/FRt4/detection_loss‚3.0902355337142944 1loss/SRqqt2/detection_loss‚2.5070478864338086 /loss/FRt3/assignment_loss‚1.466299923639449 1loss/FBt2/assignment_loss‚0.15496690726146578 0loss/FBt3/assignment_loss‚0.3497977501796413 /loss/FRt3/detection_loss‚1.7360240591187326 /loss/FBt2/detection_loss‚2.0217617670694987 /loss/FRt1/assignment_loss‚2.421123970528038 0loss/FRt4/assignment_loss‚2.9448874950408936 % _timestamp‚1.7885066661235597e+09 0loss/FRt2/assignment_loss‚3.0175433014378403 /loss/FBt3/detection_loss‚2.8540604591369627 epoch‚7 2loss/SRqqt2/assignment_loss‚1.2504544653322385 2loss/SRqqt3/assignment_loss‚0.9004229596127634 1loss/SRqqt3/detection_loss‚0.9956776797609485 _step‚744 1loss/SRqqt1/detection_loss‚2.0067825205624104 $loss/total_loss‚40.8424349230002˜H‡ýcÃ&Ý 2loss/SRqqt4/assignment_loss‚1.4151269878176125 /loss/FRt2/detection_loss‚2.5764584037842173 0loss/SRqqt4/detection_loss‚2.793613302436742 .loss/FBt4/detection_loss‚2.443286540607611 trainer/global_step‚9049 2loss/SRqqt1/assignment_loss‚1.0743721105779211 1loss/FBt1/assignment_loss‚0.27740730934488234 /loss/FBt1/detection_loss‚0.7042060270468937 _runtime‚10723.900735215 1loss/FBt4/assignment_loss‚0.18760927295983265 /loss/FRt1/detection_loss‚1.6532682895955002 /loss/FRt4/detection_loss‚3.0902355337142944 1loss/SRqqt2/detection_loss‚2.5070478864338086 /loss/FRt3/assignment_loss‚1.466299923639449 1loss/FBt2/assignment_loss‚0.15496690726146578 0loss/FBt3/assignment_loss‚0.3497977501796413 /loss/FRt3/detection_loss‚1.7360240591187326 /loss/FBt2/detection_loss‚2.0217617670694987 /loss/FRt1/assignment_loss‚2.421123970528038 0loss/FRt4/assignment_loss‚2.9448874950408936 % _timestamp‚1.7885066661235597e+09 0loss/FRt2/assignment_loss‚3.0175433014378403 /loss/FBt3/detection_loss‚2.8540604591369627 epoch‚7 2loss/SRqqt2/assignment_loss‚1.2504544653322385 2loss/SRqqt3/assignment_loss‚0.9004229596127634 1loss/SRqqt3/detection_loss‚0.9956776797609485 _step‚744 1loss/SRqqt1/detection_loss‚2.0067825205624104 $loss/total_loss‚40.8424349230002è²>?+‘Ä&‚(:† «äéÔ¯Žƒn gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.548' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.637 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tþ2”nÅ&‚(:ã «äéÔ„ûßn disk.nvme1n1.in‚ 744.57421875# disk.nvme1n1.out‚43931.33203125 proc.memory.rssMB‚3013.625 disk.sda.out‚1197.3857421875 disk.sda1.out‚0 disk.rbd0.in‚0! disk.sdb2.out‚1197.3857421875 disk.sda.in‚8.515625 disk.sda1.in‚0* proc.memory.percent‚18.393707275390625 disk.sdb.in‚3.78125 disk.sda2.in‚8.515625# disk.nvme0n1.out‚113251.5390625) proc.memory.availableMB‚ 1307.45703125 network.sent‚ 2.3700441e+07 disk.rbd0.out‚ 2.8203125 disk.dm-0.in‚ 1684.23828125 disk.md0.out‚ 1151.74609375 disk.rbd2.in‚0 cpu‚25.198849410146234 disk.sdb.out‚1197.3857421875! disk.sda2.out‚1197.3857421875 network.recv‚ 3.452115e+06( disk./.usagePercent‚75.7716338064055% disk./.usageGB‚2709.9606437683105 disk.md0p1.out‚ 1151.74609375! disk.dm-0.out‚157182.87109375$ memory_percent‚92.01991558074951 disk.sdb1.in‚0 disk.nvme0n1.in‚ 939.6640625 disk.md0.in‚ 12.296875 disk.rbd2.out‚ 3.19873046875 disk.md0p1.in‚ 12.296875 proc.cpu.threads‚17 disk.sdb2.in‚3.78125 disk.sdb1.out‚0Ë9´“Æ&‚(:ˆ ºäéÔÓ«°m gpu.0.gpu‚10.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.813' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.70325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03lÞgÇ&‚(:Ü ºäéÔ¢þ—n! disk.dm-0.out‚157479.69921875) disk./.usagePercent‚75.78385209034164% disk./.usageGB‚2710.3976287841797 disk.rbd2.in‚0 disk.sdb2.in‚3.78125 disk.sda.in‚8.515625 disk.md0p1.out‚ 1153.15625 disk.sda2.in‚8.515625# disk.nvme0n1.out‚113370.4453125 disk.rbd2.out‚ 3.19873046875 disk.nvme0n1.in‚ 939.6640625* proc.memory.percent‚18.393707275390625 proc.cpu.threads‚17 disk.md0.in‚ 12.296875 disk.rbd0.in‚0 disk.md0p1.in‚ 12.296875 disk.sdb.in‚3.78125 proc.memory.rssMB‚3013.625 network.recv‚ 3.458064e+06 disk.rbd0.out‚ 2.8994140625 disk.sda2.out‚1198.861328125 disk.nvme1n1.in‚ 744.57421875$ memory_percent‚92.02065467834473 disk.sda.out‚1198.861328125 disk.sda1.in‚0 cpu‚25.198621554902296 network.sent‚ 2.3810596e+07 disk.md0.out‚ 1153.15625 disk.sda1.out‚0 disk.sdb1.out‚0# disk.nvme1n1.out‚44109.25390625 disk.sdb.out‚1198.861328125( proc.memory.availableMB‚ 1307.3359375 disk.sdb2.out‚1198.861328125 disk.sdb1.in‚0 disk.dm-0.in‚ 1684.23828125/¹âú“È&‚(:ˆ ÉäéÔ•±ól gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.727' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.18175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00•AgÉ&‚(:Ü ÉäéÔË™Äm disk.rbd0.out‚ 2.8994140625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 1684.23828125! disk.sda2.out‚1200.8388671875' proc.memory.availableMB‚ 1307.328125 network.sent‚ 2.3815851e+07! disk.sdb2.out‚1200.8388671875 disk.sda1.in‚0 cpu‚25.198394738201436 disk.nvme0n1.in‚ 939.6640625# disk.nvme0n1.out‚113370.4453125 disk.md0.in‚ 12.296875 disk.md0.out‚ 1155.06640625 disk.rbd2.in‚0 disk.dm-0.out‚157535.8828125 proc.memory.rssMB‚3013.625 disk.rbd2.out‚ 3.26171875 disk.rbd0.in‚0 disk.sdb.in‚3.78125$ memory_percent‚92.02070236206055 disk.sdb2.in‚3.78125 disk.nvme1n1.in‚ 744.57421875 disk.sdb.out‚1200.8388671875* proc.memory.percent‚18.393707275390625 proc.cpu.threads‚17 network.recv‚ 3.46147e+06 disk.md0p1.in‚ 12.296875) disk./.usagePercent‚75.79087067333685 disk.nvme1n1.out‚ 44165.4375 disk.sda.out‚1200.8388671875 disk.sda1.out‚0 disk.sda2.in‚8.515625% disk./.usageGB‚2710.6486473083496 disk.sda.in‚8.515625 disk.md0p1.out‚ 1155.06640625Xò!á‘Ê&‚(:† ØäéÔȯðl gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚151.008' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.752 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’Þ’OË&‚(:Ä ØäéÔ¤ðìm disk.sdb1.in‚0! disk.dm-0.out‚157825.38671875 disk.sdb.in‚3.78125$ disk.nvme0n1.out‚113542.51171875 network.sent‚ 2.3821784e+07 disk.sda1.out‚0 disk.sda2.out‚ 1202.3125 disk.nvme0n1.in‚ 939.6640625) disk./.usagePercent‚75.79084219497477 disk.nvme1n1.in‚ 744.57421875$ memory_percent‚92.02375411987305% disk./.usageGB‚2710.6476287841797' proc.memory.availableMB‚ 1306.828125 disk.md0.in‚ 12.296875 disk.md0.out‚ 1156.48046875 disk.sdb2.in‚3.78125 disk.sda1.in‚0 disk.md0p1.out‚ 1156.48046875 disk.dm-0.in‚ 1684.23828125* proc.memory.percent‚18.393707275390625 network.recv‚ 3.465766e+06 disk.sda.in‚8.515625 disk.rbd2.out‚ 3.26171875 disk.rbd0.out‚ 2.96240234375 disk.md0p1.in‚ 12.296875 disk.sda2.in‚8.515625 cpu‚25.198094386375 disk.sdb2.out‚ 1202.3125 disk.sda.out‚ 1202.3125 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sdb.out‚ 1202.3125 disk.nvme1n1.out‚ 44282.875 proc.memory.rssMB‚3013.625 proc.cpu.threads‚17 disk.rbd2.in‚0\¤Ä4Ì&/ _wandbruntime‚10781  _runtime‚10781aeg³ÊÍ&Ä trainer/global_step‚9099 % _timestamp‚1.7885067234026082e+09 _runtime‚10781.180034139 _step‚745 ' lr-AdamW/pg1‚0.00018430506265960115 ' lr-AdamW/pg2‚0.00018430506265960115KªL¡ÏÎ&É % _timestamp‚1.7885067234026082e+09 _runtime‚10781.180034139 _step‚745 ' lr-AdamW/pg1‚0.00018430506265960115 ' lr-AdamW/pg2‚0.00018430506265960115 trainer/global_step‚9099騶ö4Ï&/ _wandbruntime‚10781  _runtime‚10781Ç–JwúÐ&P H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55165620 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚746 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 420780235 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 230462181 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214102016 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10384714069935 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 230488525 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 190291708 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚741319857352368 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚777111195387904 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 169181010 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚766686579788288 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 114015392 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚845888916077056 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 230462181 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚856772267760640 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚741319874391728 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚777110267413504 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 190291710 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238042112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚845888892136960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚777110267413504 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10883375623680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚751704571422303 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10423687625216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚751705496436667 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10424597596672 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚766686579788288 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 190291708 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚766686597791232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185GîÖI (Ñ&†P QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚845888916077056 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 230462181 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚856772267760640 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚741319874391728 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚777110267413504 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 190291710 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238042112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚845888892136960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚777110267413504 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10883375623680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚751704571422303 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10423687625216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚751705496436667 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10424597596672 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚766686579788288 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 190291708 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚766686597791232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55165620 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚746 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 420780235 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 230462181 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214102016 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10384714069935 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 230488525 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 190291708 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚741319857352368 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚777111195387904 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 169181010 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚766686579788288 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 114015392 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0êŠÖpÙ4Ò&/ _wandbruntime‚10782  _runtime‚10782«k°O'Ó&‹N MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 230513382 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚766768216877056 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 114027455 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚751785631304697 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 114027457 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10884768308736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚626 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚845979117841408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚741398874946912 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10425719070208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚845979141781504 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10386739318425 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚777193935947264 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚766768198874112 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 190311870 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10884541637120 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 420798880 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10425719070208 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10424809085440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚741398891986272 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚766768216877056 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚777193007959552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 190311868 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚856863910090240 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10385831343213 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 190311870 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 230487012 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 169198798 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 420825252 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 420825252تMë Ô&N EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚741398891986272 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚766768216877056 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚777193007959552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 190311868 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚856863910090240 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10385831343213 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 190311870 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 230487012 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 169198798 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 420825252 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 420825252 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 230513382 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚766768216877056 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 114027455 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚751785631304697 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 114027457 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10884768308736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚626 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚845979117841408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚741398874946912 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10425719070208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚845979141781504 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10386739318425 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚777193935947264 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚766768198874112 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 190311870 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10884541637120 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 420798880 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10425719070208 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10424809085440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615ë°[Â4Õ&/ _wandbruntime‚10782  _runtime‚10782oS;aÖ&Û _runtime‚10782.226820465 0loss/FRt4/assignment_loss‚3.7314442042951232 2loss/SRqqt4/assignment_loss‚1.5400282781767218 1loss/FBt2/assignment_loss‚0.10574057431566823 1loss/FBt4/assignment_loss‚0.11725013578067944 /loss/FRt3/detection_loss‚1.8562157356978528 % _timestamp‚1.7885067244496856e+09 _step‚748 /loss/FRt2/assignment_loss‚3.072427978515625 /loss/FRt2/detection_loss‚1.9671674823388456 /loss/FRt4/detection_loss‚2.8469716398804277 1loss/SRqqt1/detection_loss‚1.6900734039577279 0loss/FRt1/assignment_loss‚2.1582692410006667 /loss/FBt2/detection_loss‚1.9638965898438503 /loss/FBt3/detection_loss‚1.9824559220245905 epoch‚7 trainer/global_step‚9099 /loss/FBt1/detection_loss‚0.7385852546498611 0loss/FRt3/assignment_loss‚1.4007444488150733 .loss/FBt4/detection_loss‚2.201150307431817 2loss/SRqqt3/assignment_loss‚0.9365066741067436 2loss/SRqqt2/assignment_loss‚1.2242789646344525 0loss/FBt3/assignment_loss‚0.3336139762730649 1loss/SRqqt2/detection_loss‚2.5525815933942795 2loss/SRqqt1/assignment_loss‚1.4592658102109626 1loss/FBt1/assignment_loss‚0.09159667646337068 /loss/FRt1/detection_loss‚2.2669105805897964 1loss/SRqqt3/detection_loss‚0.9365677483109457 1loss/SRqqt4/detection_loss‚2.9973811758191964 %loss/total_loss‚40.17112439652735K„F@f×&à 2loss/SRqqt2/assignment_loss‚1.2242789646344525 0loss/FBt3/assignment_loss‚0.3336139762730649 1loss/SRqqt2/detection_loss‚2.5525815933942795 2loss/SRqqt1/assignment_loss‚1.4592658102109626 1loss/FBt1/assignment_loss‚0.09159667646337068 /loss/FRt1/detection_loss‚2.2669105805897964 1loss/SRqqt3/detection_loss‚0.9365677483109457 1loss/SRqqt4/detection_loss‚2.9973811758191964 %loss/total_loss‚40.17112439652735 _runtime‚10782.226820465 0loss/FRt4/assignment_loss‚3.7314442042951232 2loss/SRqqt4/assignment_loss‚1.5400282781767218 1loss/FBt2/assignment_loss‚0.10574057431566823 1loss/FBt4/assignment_loss‚0.11725013578067944 /loss/FRt3/detection_loss‚1.8562157356978528 % _timestamp‚1.7885067244496856e+09 _step‚748 /loss/FRt2/assignment_loss‚3.072427978515625 /loss/FRt2/detection_loss‚1.9671674823388456 /loss/FRt4/detection_loss‚2.8469716398804277 1loss/SRqqt1/detection_loss‚1.6900734039577279 0loss/FRt1/assignment_loss‚2.1582692410006667 /loss/FBt2/detection_loss‚1.9638965898438503 /loss/FBt3/detection_loss‚1.9824559220245905 epoch‚7 trainer/global_step‚9099 /loss/FBt1/detection_loss‚0.7385852546498611 0loss/FRt3/assignment_loss‚1.4007444488150733 .loss/FBt4/detection_loss‚2.201150307431817 2loss/SRqqt3/assignment_loss‚0.9365066741067436쨛%$“Ø&‚(:ˆ çäéÔÄÖˆm gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.305' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.07625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b-~qÙ&‚(:æ çäéÔÞ¢Êm network.recv‚ 3.469599e+06 disk.nvme0n1.in‚ 939.6640625 proc.memory.rssMB‚3013.625$ disk./.usageGB‚2710.120044708252 disk.nvme1n1.in‚ 744.57421875 disk.rbd2.out‚ 3.32470703125( proc.memory.availableMB‚ 1307.1484375 proc.cpu.threads‚17 disk.sda2.in‚8.515625$ disk.nvme0n1.out‚113657.16796875" disk.nvme1n1.out‚ 44447.1328125 disk.md0.in‚ 12.296875 disk.dm-0.in‚ 1684.23828125 disk.md0.out‚ 1158.02734375 disk.sdb2.in‚3.78125 disk.sda1.in‚0 network.sent‚ 2.3838336e+07) disk./.usagePercent‚75.77609072340074! disk.sda2.out‚1203.9208984375 disk.sda.out‚1203.9208984375 disk.sdb1.in‚0 disk.md0p1.in‚ 12.296875 disk.sdb.in‚3.78125 disk.sdb.out‚1203.9208984375* proc.memory.percent‚18.393707275390625! disk.sdb2.out‚1203.9208984375 disk.rbd0.in‚0 disk.rbd0.out‚ 2.96240234375 disk.sdb1.out‚0 disk.md0p1.out‚ 1158.02734375! disk.dm-0.out‚158104.30078125 disk.rbd2.in‚0 disk.sda.in‚8.515625 disk.sda1.out‚0$ memory_percent‚92.02179908752441 cpu‚25.197728315762905uMÖû‘Ú&‚(:† öäéÔÏÈÜk gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.204' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.551 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Oë1uÛ&‚(:ê öäéÔ¼æ¥m" disk.sdb2.out‚1205.48779296875 disk.md0p1.in‚ 12.296875 disk.md0.in‚ 12.296875 disk.md0.out‚ 1159.51953125* proc.memory.percent‚18.393707275390625 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.md0p1.out‚ 1159.51953125$ disk.nvme0n1.out‚113714.58203125" disk.nvme1n1.out‚ 44448.0390625 disk.rbd0.in‚0 disk.rbd0.out‚ 3.025390625! disk.sdb.out‚1205.48779296875 disk.rbd2.out‚ 3.32470703125$ memory_percent‚92.02172756195068% disk./.usageGB‚2710.5701484680176 disk.sda1.out‚0 disk.sdb1.in‚0 disk.dm-0.in‚ 1684.23828125" disk.sda2.out‚1205.48779296875 disk.rbd2.in‚0) proc.memory.availableMB‚ 1307.16015625 proc.memory.rssMB‚3013.625 network.recv‚ 3.482412e+06 disk.sda1.in‚0 disk.sda2.in‚8.515625 disk.nvme1n1.in‚ 744.57421875) disk./.usagePercent‚75.78867581290685 disk.sdb2.in‚3.78125 disk.sda.in‚8.515625! disk.dm-0.out‚158162.62109375 disk.sdb.in‚3.78125 disk.nvme0n1.in‚ 939.6640625 network.sent‚ 2.3939329e+07! disk.sda.out‚1205.48779296875 cpu‚25.197455741736743Oó™AÜ&‚(:’ …åéÔ™¶Åm gpu.0.gpu‚42.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=: pÝ&‚(:å …åéÔ†ÿn disk.rbd0.in‚0 disk.rbd0.out‚ 3.025390625$ memory_percent‚92.02330112457275 network.sent‚ 2.3944595e+07 disk.rbd2.out‚ 3.32470703125 disk.sdb1.out‚0) proc.memory.availableMB‚ 1306.90234375 cpu‚25.19722404944064! disk.sdb.out‚1207.51123046875" disk.sda2.out‚1207.51123046875! disk.nvme1n1.out‚ 44525.203125" disk.sdb2.out‚1207.51123046875 proc.memory.rssMB‚3013.625 disk.dm-0.in‚ 1684.23828125 disk.nvme0n1.in‚ 939.6640625 disk.sdb1.in‚0 disk.sdb2.in‚3.78125 disk.sda1.in‚0 disk.md0p1.in‚ 12.296875 network.recv‚ 3.485903e+06! disk.dm-0.out‚158354.43359375 disk.nvme1n1.in‚ 744.57421875 proc.cpu.threads‚17 disk.sdb.in‚3.78125 disk.rbd2.in‚0 disk.sda1.out‚0 disk.md0p1.out‚ 1161.4765625( disk./.usagePercent‚75.7907778786739 disk.md0.in‚ 12.296875 disk.sda.in‚8.515625* proc.memory.percent‚18.393707275390625% disk./.usageGB‚2710.6453285217285 disk.sda2.in‚8.515625$ disk.nvme0n1.out‚113829.23046875 disk.md0.out‚ 1161.4765625! disk.sda.out‚1207.51123046875!Ý ‘Þ&‚(:† ”åéÔê‚Æm gpu.0.gpu‚7.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚159.534' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.8835 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿Žå¥oß&‚(:ä ”åéÔîï²n) disk./.usagePercent‚75.79171585139338 disk.rbd2.in‚0" disk.sdb2.out‚1209.07177734375 disk.sda1.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda2.in‚8.515625) proc.memory.availableMB‚ 1307.15234375% disk./.usageGB‚2710.6788749694824 disk.md0.in‚ 12.296875 disk.sdb2.in‚3.78125 disk.rbd0.out‚ 3.025390625 network.sent‚ 2.3950612e+07 network.recv‚ 3.490239e+06 disk.dm-0.in‚ 1684.23828125! disk.dm-0.out‚158386.16015625* proc.memory.percent‚18.393707275390625" disk.sda2.out‚1209.07177734375 disk.sda1.out‚0 disk.rbd0.in‚0! disk.sdb.out‚1209.07177734375# memory_percent‚92.0217752456665 proc.cpu.threads‚17 proc.memory.rssMB‚3013.625 disk.md0p1.out‚ 1162.984375 disk.nvme1n1.in‚ 744.57421875 disk.rbd2.out‚ 3.32470703125 disk.sda.in‚8.515625 disk.md0p1.in‚ 12.296875 cpu‚25.19699681798132 disk.nvme0n1.in‚ 939.6640625$ disk.nvme0n1.out‚113829.33984375" disk.nvme1n1.out‚ 44556.8203125 disk.md0.out‚ 1162.984375! disk.sda.out‚1209.07177734375 disk.sdb.in‚3.781253¡–4à&/ _wandbruntime‚10840  _runtime‚10840ÍI{Êá&Ä ' lr-AdamW/pg2‚0.00017856668712592052 trainer/global_step‚9149 % _timestamp‚1.7885067826097267e+09 _runtime‚10840.386956721 _step‚749 ' lr-AdamW/pg1‚0.00017856668712592052ï)ÞÏâ&É trainer/global_step‚9149 % _timestamp‚1.7885067826097267e+09 _runtime‚10840.386956721 _step‚749 ' lr-AdamW/pg1‚0.00017856668712592052 ' lr-AdamW/pg2‚0.00017856668712592052íü!“4ã&/ _wandbruntime‚10840  _runtime‚10840›åÏÎ(ä&P ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚770767552082432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10480685021696 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 231707083 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10480685021696 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10479775050240 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 231707083 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55457129 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10479775050240 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚861342397263360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚755711328404611 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚781247309129728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚745270737079568 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 % _timestamp‚1.7885067826107843e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55456492 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚745270754118928 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 191299708 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚850400201299456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 231733427 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚639 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 231733427 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10941960890368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 423033137 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚781248237104128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259013632 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚781247309129728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 170075771 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 423006791 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚850400177359360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10441499300047 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 191299710 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 191299708 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚755712253418975 _runtime‚10840.388077166Âý°Ñå&†P JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚781247309129728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚745270737079568 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 % _timestamp‚1.7885067826107843e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55456492 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚745270754118928 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 191299708 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚850400201299456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 231733427 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚639 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 231733427 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10941960890368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 423033137 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚781248237104128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259013632 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚781247309129728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 170075771 \ODeviceStatsMonitÉññ;or.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 423006791 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚850400177359360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10441499300047 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 191299710 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 191299708 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚755712253418975 _runtime‚10840.388077166 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚770767552082432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10480685021696 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 231707083 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10480685021696 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10479775050240 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 231707083 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55457129 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10479775050240 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚861342397263360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚755711328404611 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004î©£i¯4æ&/ _wandbruntime‚10841  _runtime‚10841h´Ë’'ç&‹N S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚781330049675776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚770849171168256 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚745349771713472 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10480896510464 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 170093629 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 191319870 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10943132149760 _step‚751 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55462922 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚633 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 191319868 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 % _timestamp‚1.7885067837936826e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 231758284 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 170092994 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10442616573533 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚770849153165312 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10480896510464 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚755791463272433 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 231731914 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 trainer/global_step‚9149 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚861433785825280 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 191319868 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 _runtime‚10841.575166455 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 423078154 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 423078154 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 114630705 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 423051782 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10441708598321 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10481806495232 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚635 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55462289 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚850490403063808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚770849171168256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 191319870 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚861433535213568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10943358821376 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 191319870 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10943132149760 _step‚751 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55462922 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚633 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 191319868 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 % _timestamp‚1.7885067837936826e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 231758284 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 170092994 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10442616573533 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚770849153165312 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10480896510464 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚755791463272433 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 231731914 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 trainer/global_step‚9149 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚861433785825280 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 191319868 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 _runtime‚10841.575166455 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 423078154 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 423078154 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 114630705 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 423051782 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10441708598321 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10481806495232 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚635 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55462289 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚850490403063808ïNðS4é&/ _wandbruntime‚10841  _runtime‚10841"À±[ê&Õ 2loss/SRqqt3/assignment_loss‚1.0304203909119158 1loss/SRqqt3/detection_loss‚0.9980328349795725 /loss/FBt1/detection_loss‚0.6271195537255456 _step‚752 1loss/FBt4/assignment_loss‚0.34162687039703443 /loss/FRt2/detection_loss‚2.6044999593321014 0loss/SRqqt2/detection_loss‚2.270613842540317 -loss/FBt2/detection_loss‚2.40344299511476 epoch‚7 0loss/FRt1/assignment_loss‚2.2470643261204595 2loss/SRqqt2/assignment_loss‚1.5134280421048902 /loss/FRt3/detection_loss‚1.9032701197410784 /loss/FRt2/assignment_loss‚3.099743203205221 .loss/FBt4/detection_loss‚2.370479447974099 /loss/FBt3/detection_loss‚2.2187701788815586 % _timestamp‚1.7885067837996898e+09 1loss/SRqqt1/detection_loss‚2.0845072139864382 $loss/total_loss‚41.6013451158475 0loss/SRqqt4/detection_loss‚2.598899858338492 trainer/global_step‚9149 0loss/FBt2/assignment_loss‚0.3052402160617269 2loss/SRqqt1/assignment_loss‚1.2717239191026792 2loss/SRqqt4/assignment_loss‚0.9048264492303133 1loss/FBt1/assignment_loss‚0.09632316157265905 .loss/FRt4/detection_loss‚3.519741303390927 _runtime‚10841.577024604 1loss/FBt3/assignment_loss‚0.08535275877379304 /loss/FRt1/detection_loss‚1.5888407141730725 0loss/FRt3/assignment_loss‚1.7627802645006487 0loss/FRt4/assignment_loss‚3.7545974916881986<™¹`ë&Ú 0loss/FBt2/assignment_loss‚0.3052402160617269 2loss/SRqqt1/assignment_loss‚1.2717239191026792 2loss/SRqqt4/assignment_loss‚0.9048264492303133 1loss/FBt1/assignment_loss‚0.09632316157265905 .loss/FRt4/detection_loss‚3.519741303390927 _runtime‚10841.577024604 1loss/FBt3/assignment_loss‚0.08535275877379304 /loss/FRt1/detection_loss‚1.5888407141730725 0loss/FRt3/assignment_loss‚1.7627802645006487 0loss/FRt4/assignment_loss‚3.7545974916881986 2loss/SRqqt3/assignment_loss‚1.0304203909119158 1loss/SRqqt3/detection_loss‚0.9980328349795725 /loss/FBt1/detection_loss‚0.6271195537255456 _step‚752 1loss/FBt4/assignment_loss‚0.34162687039703443 /loss/FRt2/detection_loss‚2.6044999593321014 0loss/SRqqt2/detection_loss‚2.270613842540317 -loss/FBt2/detection_loss‚2.40344299511476 epoch‚7 0loss/FRt1/assignment_loss‚2.2470643261204595 2loss/SRqqt2/assignment_loss‚1.5134280421048902 /loss/FRt3/detection_loss‚1.9032701197410784 /loss/FRt2/assignment_loss‚3.099743203205221 .loss/FBt4/detection_loss‚2.370479447974099 /loss/FBt3/detection_loss‚2.2187701788815586 % _timestamp‚1.7885067837996898e+09 1loss/SRqqt1/detection_loss‚2.0845072139864382 $loss/total_loss‚41.6013451158475 0loss/SRqqt4/detection_loss‚2.598899858338492 trainer/global_step‚9149ðr˜‘ì&‚(:† £åéÔò™žn gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚121.536' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.384 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>­÷hí&‚(:Ý £åéÔ£²†o disk.rbd2.out‚ 3.32470703125 disk.sda1.in‚0 disk.dm-0.in‚ 1684.23828125 disk.dm-0.out‚ 158587.53125" disk.sda2.out‚1210.58642578125( proc.memory.availableMB‚ 1306.8671875! disk.sda.out‚1210.58642578125 network.recv‚ 3.494485e+06( disk./.usagePercent‚75.7978465921192 disk.rbd2.in‚0 disk.sda.in‚8.515625 disk.sdb1.in‚0 disk.nvme0n1.in‚ 939.6640625# disk.nvme1n1.out‚44643.54296875% disk./.usageGB‚2710.8981399536133 disk.rbd0.out‚ 3.025390625 proc.memory.rssMB‚3013.625 disk.sda1.out‚0 disk.md0.in‚ 12.296875 disk.sdb1.out‚0 disk.md0p1.out‚ 1164.4375$ memory_percent‚92.02351570129395! disk.sdb.out‚1210.58642578125 disk.nvme1n1.in‚ 744.57421875 disk.sdb2.in‚3.78125 disk.rbd0.in‚0$ disk.nvme0n1.out‚113943.98828125 cpu‚25.19676781740266 proc.cpu.threads‚17 disk.md0.out‚ 1164.4375 disk.md0p1.in‚ 12.296875* proc.memory.percent‚18.393707275390625 network.sent‚ 2.3954453e+07" disk.sdb2.out‚1210.58642578125 disk.sdb.in‚3.78125 disk.sda2.in‚8.515625›ÀÉ’î&‚(:‡ ²åéÔþÇþl gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.219' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.55475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æþ°`ï&‚(:Õ ²åéÔþó‚n* proc.memory.percent‚18.393707275390625 network.recv‚ 3.500145e+06) disk./.usagePercent‚75.80481632459069 disk.sdb2.in‚3.78125 disk.rbd0.out‚ 3.025390625 disk.md0p1.out‚ 1166.3671875! disk.dm-0.out‚158815.96484375 disk.sda2.out‚ 1212.58203125 proc.memory.rssMB‚3013.625 cpu‚25.196547574757826 disk.sdb2.out‚ 1212.58203125$ memory_percent‚92.02499389648438 proc.cpu.threads‚17 disk.md0.in‚ 12.296875 disk.md0.out‚ 1166.3671875 disk.sdb1.out‚0 disk.dm-0.in‚ 1684.23828125 disk.sdb.out‚ 1212.58203125 network.sent‚ 2.4064473e+07! disk.nvme0n1.out‚ 114116.03125 disk.nvme1n1.in‚ 744.57421875 disk.sda.out‚ 1212.58203125 disk.sda1.out‚0 disk.rbd0.in‚0% disk./.usageGB‚2711.1474113464355 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sdb1.in‚0 disk.nvme0n1.in‚ 939.6640625 disk.sda.in‚8.515625 disk.md0p1.in‚ 12.296875 disk.sda2.in‚8.515625# disk.nvme1n1.out‚44699.93359375 disk.rbd2.out‚ 3.32470703125 disk.sdb.in‚3.78125$ proc.memory.availableMB‚1306.625î’,“ð&‚(:ˆ ÁåéÔ™“÷k gpu.0.gpu‚48.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.129' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.03225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 “kbeñ&‚(:Ú ÁåéÔüƒ’m disk.sdb1.in‚0 disk.sdb.out‚1214.193359375! disk.nvme0n1.out‚ 114116.03125 disk.sda.in‚8.515625 disk.sda1.in‚0 disk.nvme1n1.in‚ 744.57421875 disk.md0.out‚ 1167.9140625 disk.sdb2.in‚3.78125* proc.memory.percent‚18.393707275390625 disk.md0p1.in‚ 12.296875 network.recv‚ 3.502774e+06) disk./.usagePercent‚75.80481653791176 disk.sda1.out‚0 disk.sdb1.out‚0$ memory_percent‚92.02494621276855 cpu‚25.196304317957576 disk.sda2.out‚1214.193359375 proc.memory.rssMB‚3013.625 proc.cpu.threads‚17 disk.rbd2.out‚ 3.32470703125 disk.sdb2.out‚1214.193359375 disk.md0p1.out‚ 1167.9140625 disk.sdb.in‚3.78125 disk.sda2.in‚8.515625 disk.nvme0n1.in‚ 939.6640625 disk.sda.out‚1214.193359375# disk.nvme1n1.out‚44817.25390625 disk.rbd0.in‚0 disk.rbd0.out‚ 3.025390625 disk.md0.in‚ 12.296875# disk./.usageGB‚2711.14741897583 disk.dm-0.in‚ 1684.23828125! disk.dm-0.out‚158933.28515625( proc.memory.availableMB‚ 1306.6328125 disk.rbd2.in‚0 network.sent‚ 2.406661e+07ª5 Ðò&‚(:… ÐåéÔ‘À¶m gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RÕŸó&‚(:Ô ÐåéÔþݦn# disk.nvme0n1.out‚114288.0703125 proc.memory.rssMB‚3013.625 proc.cpu.threads‚17 disk.dm-0.in‚ 1684.23828125# disk.nvme1n1.out‚45048.36328125 disk.rbd2.out‚ 3.32470703125 disk.sda1.in‚0* proc.memory.percent‚18.393707275390625 network.recv‚ 3.507661e+06 disk.md0.out‚ 1169.171875 disk.sdb2.out‚ 1215.5078125( disk./.usagePercent‚75.8048166445723 disk.sdb1.out‚0 disk.sdb.out‚ 1215.5078125 disk.nvme0n1.in‚ 939.6640625 disk.rbd0.out‚ 3.025390625 disk.md0p1.out‚ 1169.171875 disk.sda2.in‚8.515625 disk.sdb2.in‚3.78125 disk.rbd0.in‚0) proc.memory.availableMB‚ 1306.88671875 disk.md0p1.in‚ 12.296875 disk.rbd2.in‚0# memory_percent‚92.0233964920044 disk.sdb1.in‚0! disk.dm-0.out‚159336.43359375 disk.sda1.out‚0 network.sent‚ 2.4073036e+07% disk./.usageGB‚2711.1474227905273 disk.nvˆšCÀme1n1.in‚ 744.57421875 disk.md0.in‚ 12.296875 disk.sda.in‚8.515625 disk.sda.out‚ 1215.5078125 cpu‚25.196051839790325 disk.sdb.in‚3.78125 disk.sda2.out‚ 1215.5078125Ú&0¿4ô&/ _wandbruntime‚10900  _runtime‚10900_IǧÊõ&Ä % _timestamp‚1.7885068423125129e+09 _runtime‚10900.089756699 _step‚753 ' lr-AdamW/pg1‚0.00017290075303285837 ' lr-AdamW/pg2‚0.00017290075303285837 trainer/global_step‚9199/ÿHlÏö&É trainer/global_step‚9199 % _timestamp‚1.7885068423125129e+09 _runtime‚10900.089756699 _step‚753 ' lr-AdamW/pg1‚0.00017290075303285837 ' lr-AdamW/pg2‚0.00017290075303285837ñ.4:4÷&/ _wandbruntime‚10900  _runtime‚10900a½Íj(ø&P \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11000821126656 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚759719010401283 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 232951985 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10536772446720 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 192307708 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 232978329 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚854911486521856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10535862475264 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚865912307648512 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 115221890 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11000594441728 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚9199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55748218 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚865912057023488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚749221633846128 _runtime‚10900.094931873 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 192307710 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10496468580151 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚774848488370688 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚785385278820352 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚774848506373632 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10497376555155 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚622 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚774848506373632 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 % _timestamp‚1.7885068423136716e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚759718085386919 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚785384350845952 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10535862475264 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 232951985 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 192307708 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 232978329 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 192307710 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 425286039 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚774848488370688 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10536772446720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55748840 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚854911462581760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚785384350845952 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 170970732 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 170970108M8¬/ (ù&†P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10497376555155 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚622 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚774848506373632 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 % _timestamp‚1.7885068423136716e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚759718085386919 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚785384350845952 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10535862475264 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 232951985 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 192307708 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 232978329 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 192307710 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 425286039 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚774848488370688 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10536772446720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55748840 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚854911462581760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚785384350845952 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 170970732 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 170970108 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11000821126656 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚759719010401283 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 232951985 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10536772446720 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 192307708 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 232978329 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚854911486521856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10535862475264 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚865912307648512 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 115221890 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11000594441728 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚9199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55748218 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚865912057023488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚749221633846128 _runtime‚10900.094931873 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 192307710 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10496468580151 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚774848488370688 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚785385278820352 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚774848506373632 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 115233957 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚749300651440672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 232976816 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231737344 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11002009888256 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10498493828641 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 425331056 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11001802091008 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885068433652174e+09 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚785468019379712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 170988626 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚774930125459456 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 192327868 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚774930107456512 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚785467091392000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10536983935488 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 192327870 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 425304684 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10537893920256 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 192327868 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚855001712226304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55754669 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 425304684 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 233003186 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚749300634401312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚785468019379712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 207797248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚866003722114560 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚10901.143044338 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 trainer/global_step‚9199 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 233003186 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 232976816 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10537893920256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚774930125459456 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 192327870 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 425331056 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 170988005 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚774930107456512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚621 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55754050K'aï¡ü&N H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚774930125459456 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 192327868 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚774930107456512 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚785467091392000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10536983935488 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 192327870 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freedPÔL‡u!‚ 425304684 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10537893920256 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 192327868 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚855001712226304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55754669 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 425304684 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 233003186 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚749300634401312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚785468019379712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 207797248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚866003722114560 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚10901.143044338 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 trainer/global_step‚9199 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 233003186 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 232976816 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10537893920256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚774930125459456 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 192327870 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 425331056 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 170988005 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚774930107456512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚621 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55754050 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 115233957 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚749300651440672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 232976816 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231737344 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11002009888256 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10498493828641 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 425331056 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11001802091008 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885068433652174e+09 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚785468019379712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 170988626óœå‘z4ý&/ _wandbruntime‚10901  _runtime‚109019.´ê\þ&Ö 2loss/SRqqt1/assignment_loss‚1.5352027967572213 .loss/FBt2/detection_loss‚2.585947059922748 epoch‚7 _step‚756 0loss/FRt2/assignment_loss‚3.0147161334753036 0loss/FRt3/assignment_loss‚1.4851298166645899 /loss/FRt4/detection_loss‚3.9665011167526245 .loss/FBt4/detection_loss‚2.432669992248217 _runtime‚10901.144872407 2loss/SRqqt2/assignment_loss‚1.5901250410079957 2loss/SRqqt3/assignment_loss‚1.0585291845848228 1loss/FBt1/assignment_loss‚0.13589569728084047 /loss/FBt1/detection_loss‚0.7454139532678774 1loss/FBt3/assignment_loss‚0.12949515032848716 1loss/SRqqt1/detection_loss‚1.9258322465419768 .loss/FRt2/detection_loss‚2.352757232136686 /loss/FRt4/assignment_loss‚3.297211669385433 .loss/FRt1/detection_loss‚1.828986432847504 1loss/SRqqt3/detection_loss‚1.1164760170582713 %loss/total_loss‚42.12950602344207 0loss/FRt1/assignment_loss‚2.2163650519920117 1loss/FBt4/assignment_loss‚0.08185854620678394 % _timestamp‚1.7885068433678901e+09 2loss/SRqqt4/assignment_loss‚1.1594120732375555 0loss/FBt2/assignment_loss‚0.3257929886238142 /loss/FRt3/detection_loss‚1.7741229067108146 .loss/FBt3/detection_loss‚2.161372509869662 0loss/SRqqt2/detection_loss‚2.362323079109192 1loss/SRqqt4/detection_loss‚2.8473693274316334 trainer/global_step‚9199&Ž~aÿ&Û .loss/FBt3/detection_loss‚2.161372509869662 0loss/SRqqt2/detection_loss‚2.362323079109192 1loss/SRqqt4/detection_loss‚2.8473693274316334 trainer/global_step‚9199 2loss/SRqqt1/assignment_loss‚1.5352027967572213 .loss/FBt2/detection_loss‚2.585947059922748 epoch‚7 _step‚756 0loss/FRt2/assignment_loss‚3.0147161334753036 0loss/FRt3/assignment_loss‚1.4851298166645899 /loss/FRt4/detection_loss‚3.9665011167526245 .loss/FBt4/detection_loss‚2.432669992248217 _runtime‚10901.144872407 2loss/SRqqt2/assignment_loss‚1.5901250410079957 2loss/SRqqt3/assignment_loss‚1.0585291845848228 1loss/FBt1/assignment_loss‚0.13589569728084047 /loss/FBt1/detection_loss‚0.7454139532678774 1loss/FBt3/assignment_loss‚0.12949515032848716 1loss/SRqqt1/detection_loss‚1.9258322465419768 .loss/FRt2/detection_loss‚2.352757232136686 /loss/FRt4/assignment_loss‚3.297211669385433 .loss/FRt1/detection_loss‚1.828986432847504 1loss/SRqqt3/detection_loss‚1.1164760170582713 %loss/total_loss‚42.12950602344207 0loss/FRt1/assignment_loss‚2.2163650519920117 1loss/FBt4/assignment_loss‚0.08185854620678394 % _timestamp‚1.7885068433678901e+09 2loss/SRqqt4/assignment_loss‚1.1594120732375555 0loss/FBt2/assignment_loss‚0.3257929886238142 /loss/FRt3/detection_loss‚1.7741229067108146ô³úá’€'‚(:‡ ßåéÔ¶ï…m gpu.0.gpu‚41.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚132.122' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.0305 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ?µì\'‚(:Ñ ßåéÔ»¡ºm network.recv‚ 3.518215e+06( disk./.usagePercent‚75.7900911981681 disk.sda2.in‚8.515625 disk.md0.out‚ 1170.734375$ memory_percent‚92.02532768249512 disk.rbd2.in‚0 disk.sdb2.out‚ 1217.1328125 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sda2.out‚ 1217.1328125 cpu‚25.195809788043498 disk.sdb2.in‚3.78125 disk.sdb.in‚3.78125 disk.sda.in‚8.515625 disk.dm-0.in‚ 1684.23828125 disk.dm-0.out‚ 159385.234375 disk.nvme0n1.in‚ 939.6640625 disk.nvme1n1.in‚ 744.57421875 disk.sda.out‚ 1217.1328125# disk.nvme0n1.out‚114288.0703125% disk./.usageGB‚2710.6207695007324 disk.rbd0.out‚ 3.025390625 disk.sdb1.out‚0" disk.nvme1n1.out‚ 45097.1640625( proc.memory.availableMB‚ 1306.5703125 proc.cpu.threads‚17 network.sent‚ 2.4082057e+07 disk.sda1.in‚0 disk.md0p1.out‚ 1170.734375 proc.memory.rssMB‚3013.625 disk.rbd2.out‚ 3.32470703125 disk.md0p1.in‚ 12.296875 disk.sdb.out‚ 1217.1328125 disk.md0.in‚ 12.296875* proc.memory.percent‚18.393707275390625¹•af‚'‚(:Û îåéԆȴn disk.nvme1n1.in‚ 744.57421875 disk.sda.in‚8.515625 disk.md0p1.out‚ 1172.4765625$ disk.nvme0n1.out‚114517.69140625 disk.md0.in‚ 12.296875! disk.sdb2.out‚1218.9326171875 disk.sda1.out‚0' proc.memory.availableMB‚ 1306.953125 disk.rbd0.in‚0 disk.sda1.in‚0 proc.memory.rssMB‚3013.625 cpu‚25.195582475510886 network.recv‚ 3.52488e+06 disk.sdb1.out‚0# disk.nvme1n1.out‚45097.55859375 disk.rbd2.out‚ 3.32470703125* proc.memory.percent‚18.393707275390625( disk./.usagePercent‚75.7900908781865 disk.md0p1.in‚ 12.296875 disk.dm-0.in‚ 1684.23828125 disk.dm-0.out‚ 159615.25 disk.sdb.in‚3.78125 disk.sdb2.in‚3.78125 disk.sda.out‚1218.9326171875$ memory_percent‚92.02299118041992 disk.sda2.in‚8.515625 disk.rbd2.in‚0 proc.cpu.threads‚17 network.sent‚ 2.4192324e+07 disk.sdb1.in‚0 disk.sdb.out‚1218.9326171875 disk.md0.out‚ 1172.4765625% disk./.usageGB‚2710.6207580566406 disk.rbd0.out‚ 3.025390625! disk.sda2.out‚1218.9326171875 disk.nvme0n1.in‚ 939.6640625'»²•“ƒ'‚(:ˆ îåéÔÕ¤¶š gpu.0.gpu‚8.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.235' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.80875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[d)ê‘„'‚(:† ýåéÔ‘ðâk gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚141.704' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.426 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:t…'‚(:é ýåéÔ›´¤t network.recv‚ 3.528577e+06" disk.sda2.out‚1220.83349609375! disk.sda.out‚1220.83349609375 disk.sda2.in‚8.515625" disk.nvme0n1.out‚ 114594.703125" disk.sdb2.out‚1220.83349609375 disk.sda1.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 1684.23828125 disk.sdb.in‚3.78125$ memory_percent‚92.02306270599365 disk.rbd2.in‚0% disk./.usageGB‚2710.6461219787598# disk.nvme1n1.out‚45098.34765625 disk.sdb2.in‚3.78125 disk.rbd0.out‚ 3.025390625 disk.md0p1.out‚ 1174.30078125! disk.sdb.out‚1220.83349609375 cpu‚25.195377821236946 disk.nvme1n1.in‚ 744.57421875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sdb1.in‚0) proc.memory.availableMB‚ 1306.94140625* proc.memory.percent‚18.393707275390625 proc.cpu.threads‚17 disk.rbd2.out‚ 3.32470703125 network.sent‚ 2.4197576e+07 disk.md0.out‚ 1174.30078125 disk.sda.in‚8.515625 proc.memory.rssMB‚3013.625! disk.dm-0.out‚159693.05078125 disk.nvme0n1.in‚ 939.6640625 disk.md0p1.in‚ 12.296875) disk./.usagePercent‚75.79080006406458 disk.md0.in‚ 12.296875ºfö„’†'‚(:‡ ŒæéÔ“Èÿl gpu.0.gpu‚13.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚118.142' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.5355 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–ÇSc‡'‚(:Ø ŒæéÔÎÑÝm disk.nvme1n1.out‚ 45098.59375) disk./.usagePercent‚75.75951951648818 disk.md0p1.in‚ 12.296875 disk.sdb2.out‚ 1222.42578125 disk.sdb.out‚ 1222.42578125$ disk.nvme0n1.out‚114824.16796875 disk.sdb2.in‚3.78125% disk./.usageGB‚2709.5273780822754 disk.sda.in‚8.515625 disk.sda.out‚ 1222.42578125 disk.sda1.in‚0 disk.nvme0n1.in‚ 939.6640625 disk.md0.in‚ 12.296875 proc.cpu.threads‚17 network.sent‚ 2.4203633e+07 disk.dm-0.in‚ 1684.23828125 disk.sdb.in‚3.78125' proc.memory.availableMB‚ 1306.828125 cpu‚25.195096681997214 network.recv‚ 3.532931e+06 disk.rbd0.in‚0 disk.rbd0.out‚ 3.025390625 disk.md0p1.out‚ 1175.8203125 disk.sda2.out‚ 1222.42578125 disk.nvme1n1.in‚ 744.57421875 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sdb1.in‚0 disk.sdb1.out‚0! disk.dm-0.out‚159922.76171875 disk.md0.out‚ 1175.8203125$ memory_percent‚92.02375411987305* proc.memory.percent‚18.393707275390625 disk.sda2.in‚8.515625 disk.rbd2.out‚ 3.32470703125 proc.memory.rssMB‚3013.625¯Ø64ˆ'/ _wandbruntime‚10956  _runtime‚10956=H8ɉ'à ' lr-AdamW/pg1‚0.00016730844115258773 ' lr-AdamW/pg2‚0.00016730844115258773 trainer/global_step‚9249 $ _timestamp‚1.788506898930468e+09 _runtime‚10956.707730672 _step‚757hõ˜åΊ'È trainer/global_step‚9249 $ _timestamp‚1.788506898930468e+09 _runtime‚10956.707730672 _step‚757 ' lr-AdamW/pg1‚0.00016730844115258773 ' lr-AdamW/pg2‚0.00016730844115258773õ[ʇ4‹'/ _wandbruntime‚10956  _runtime‚10956®ý(Œ'€P RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 171865611 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚10956.708840397 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10591949900288 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚859422747804160 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚778929442661888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚789521392562176 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚859422771744256 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10553253810263 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚753172496533968 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10592859871744 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11059022839296 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10552345835259 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 234196887 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚778929460664832 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚870482027559936 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10591949900288 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 171864986 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56040469 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚753172513573328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 $ _timestamp‚1.788506898931548e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚789522320536576 trainer/global_step‚9249 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚789521392562176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 193315708 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 193315710 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚758 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 427538941 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 115825142 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006vÔ‡6'…P ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 427512595 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚778929460664832 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 193315708 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 193315710 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11059255815680 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 234223231 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 $ _timestamp‚1.788506898931548e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚789522320536576 trainer/global_step‚9249 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚789521392562176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 193315708 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 193315710 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚758 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 427538941 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 115825142 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 171865611 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚10956.708840397 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10591949900288 BÕ_train_batch_start/inactive_split.small_pool.freed‚56039846 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 234196887 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚870481770643456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚859422747804160 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚778929442661888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚789521392562176 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚859422771744256 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10553253810263 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚753172496533968 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10592859871744 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11059022839296 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10552345835259 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 234196887 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚778929460664832 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚870482027559936 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10591949900288 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 171864986 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56040469 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚753172513573328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 171882860 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚763804977237049 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10593981345280 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 427583958 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 234248088 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10593981345280 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 115837207 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11060207333888 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 427583958 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 234248088 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11060429811200 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚753251514128512 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 115837205 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚870573180842496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚870573427259904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 _runtime‚10957.647029859 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚789605061095936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 427557586 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚763805902251621 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 193335870 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚779011079750656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚617 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚859512997448704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 171883477 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 193335870 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚779011061747712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10554371083749 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚789604133108224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0h|Á ''N PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 115837205 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚870573180842496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚870573427259904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 _runtime‚10957.647029859 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚789605061095936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 427557586 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚763805902251621 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 193335870 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚779011079750656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚617 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚859512997448704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 171883477 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 193335870 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚779011061747712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10554371083749 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚789604133108224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚779011079750656 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 171882860 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚763804977237049 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10593981345280 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 427583958 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 234248088 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10593981345280 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 115837207 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11060207333888 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 427583958 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 234248088 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11060429811200 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚753251514128512 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312÷cÒ4‘'/ _wandbruntime‚10957  _runtime‚10957 lƒ¼_’'Ù 0loss/FRt2/assignment_loss‚2.7687124321537633 2loss/SRqqt2/assignment_loss‚1.1265184732037596 0loss/FBt1/assignment_loss‚0.1410730516805592 1loss/SRqqt4/detection_loss‚2.2970055723801637 _runtime‚10957.648538605 1loss/SRqqt3/detection_loss‚0.9337735312001314 _step‚760 0loss/FRt3/assignment_loss‚1.7250715624541044 1loss/SRqqt3/assignment_loss‚1.085397287689375 2loss/SRqqt4/assignment_loss‚1.6377923016746838 /loss/FRt1/detection_loss‚1.9046653585255626 /loss/FBt1/detection_loss‚0.6001730902718185 epoch‚7 0loss/SRqqt2/detection_loss‚2.644802875816822 /loss/FBt2/detection_loss‚2.4698114720257847 0loss/FRt4/assignment_loss‚3.2513973765513478 1loss/FBt4/assignment_loss‚0.11778192885300313 .loss/FRt4/detection_loss‚2.896399900317192 /loss/SRqqt1/detection_loss‚1.99919155624605 /loss/FBt3/detection_loss‚2.6151741981506347 /loss/FBt4/detection_loss‚3.0267484575510024 1loss/FBt2/assignment_loss‚0.16883028718769888 3loss/FBt3/assignment_loss‚0.0027647167442864883 &loss/total_loss‚41.151123876503945 % _timestamp‚1.7885068998715482e+09 0loss/FRt1/assignment_loss‚2.2596548142887296 2loss/SRqqt1/assignment_loss‚1.2221732214093208 .loss/FRt3/detection_loss‚1.639896742941346 .loss/FRt2/detection_loss‚2.616313667186806 trainer/global_step‚9249!.Fd“'Þ &loss/total_loss‚41.151123876503945 % _timestamp‚1.7885068998715482e+09 0loss/FRt1/assignment_loss‚2.2596548142887296 2loss/SRqqt1/assignment_loss‚1.2221732214093208 .loss/FRt3/detection_loss‚1.639896742941346 .loss/FRt2/detection_loss‚2.616313667186806 trainer/global_step‚9249 0loss/FRt2/assignment_loss‚2.7687124321537633 2loss/SRqqt2/assignment_loss‚1.1265184732037596 0loss/FBt1/assignment_loss‚0.1410730516805592 1loss/SRqqt4/detection_loss‚2.2970055723801637 _runtime‚10957.648538605 1loss/SRqqt3/detection_loss‚0.9337735312001314 _step‚760 0loss/FRt3/assignment_loss‚1.7250715624541044 1loss/SRqqt3/assignment_loss‚1.085397287689375 2loss/SRqqt4/assignment_loss‚1.6377923016746838 /loss/FRt1/detection_loss‚1.9046653585255626 /loss/FBt1/detection_loss‚0.6001730902718185 epoch‚7 0loss/SRqqt2/detection_loss‚2.644802875816822 /loss/FBt2/detection_loss‚2.4698114720257847 0loss/FRt4/assignment_loss‚3.2513973765513478 1loss/FBt4/assignment_loss‚0.11778192885300313 .loss/FRt4/detection_loss‚2.896399900317192 /loss/SRqqt1/detection_loss‚1.99919155624605 /loss/FBt3/detection_loss‚2.6151741981506347 /loss/FBt4/detection_loss‚3.0267484575510024 1loss/FBt2/assignment_loss‚0.16883028718769888 3loss/FBt3/assignment_loss‚0.0027647167442864883ø s[ ‘”'‚(:† ›æéÔú¹íl gpu.0.gpu‚17.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.844' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.961 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dßp•'‚(:å ›æéÔ¡€n% disk./.usageGB‚2709.8406829833984 disk.sda1.out‚0 disk.rbd0.out‚ 3.025390625" disk.sda2.out‚1224.41162109375 disk.rbd2.out‚ 3.32470703125 disk.sdb2.in‚3.78125 proc.memory.rssMB‚3013.625 cpu‚25.194851823711552 disk.md0.out‚ 1177.73046875 network.recv‚ 3.537002e+06 disk.sda.in‚8.515625! disk.dm-0.out‚160010.25390625 disk.sdb.in‚3.78125 disk.sdb1.in‚0 disk.sdb1.out‚0& proc.memory.availableMB‚ 1306.53125 network.sent‚ 2.4220077e+07! disk.sda.out‚1224.41162109375! disk.sdb.out‚1224.41162109375! disk.nvme0n1.out‚ 114911.46875 disk.rbd2.in‚0" disk.sdb2.out‚1224.41162109375$ memory_percent‚92.02556610107422 disk.rbd0.in‚0 disk.md0p1.in‚ 12.296875 disk.md0p1.out‚ 1177.73046875 disk.nvme0n1.in‚ 939.6640625 disk.nvme1n1.in‚ 744.57421875 disk.md0.in‚ 12.296875) disk./.usagePercent‚75.76827965265187 disk.dm-0.in‚ 1684.23828125# disk.nvme1n1.out‚45098.78515625 disk.sda1.in‚0 disk.sda2.in‚8.515625* proc.memory.percent‚18.393707275390625 proc.cpu.threads‚17ŽñÕÕ’–'‚(:‡ ªæéÔÄŒm gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚133.958' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.4895 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ào5„i—'‚(:Þ ªæéÔˆüêm disk.md0.out‚ 1179.02734375! disk.sdb2.out‚1225.7705078125 disk.sdb1.in‚0 disk.dm-0.in‚ 1684.23828125 network.sent‚ 2.4317987e+07) disk./.usagePercent‚75.77528639632801 disk.md0.in‚ 12.296875 disk.md0p1.in‚ 12.296875$ memory_percent‚92.02706813812256 proc.memory.rssMB‚3013.625 cpu‚25.194609791181847 disk.nvme1n1.in‚ 744.57421875 disk.rbd2.in‚0 disk.sdb2.in‚3.78125 disk.sda1.out‚0! disk.nvme1n1.out‚ 45099.578125 disk.sda.out‚1225.7705078125) proc.memory.availableMB‚ 1306.28515625 network.recv‚ 3.54389e+06 disk.sda.in‚8.515625 disk.sda1.in‚0 disk.rbd0.out‚ 3.025390625 disk.nvme0n1.in‚ 939.6640625! disk.nvme0n1.out‚ 115097.09375* proc.memory.percent‚18.393707275390625$ disk./.usageGB‚2710.091278076172 disk.rbd2.out‚ 3.32470703125 disk.md0p1.out‚ 1179.02734375 disk.dm-0.out‚ 160196.671875 disk.sdb.out‚1225.7705078125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sdb.in‚3.78125 disk.sda2.in‚8.515625! disk.sda2.out‚1225.7705078125–ƒlo˜'‚(:… ¹æéÔµÛºm gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù»Ãô™'‚(:â ¹æéÔ§„¥n disk.md0.in‚ 12.296875 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 3.025390625 disk.rbd2.out‚ 3.32470703125" disk.sdb2.out‚1227.50732421875 disk.nvme0n1.out‚ 115097.125) disk./.usagePercent‚75.77528074331981 disk.sdb.in‚3.78125 disk.md0.out‚ 1180.69140625& proc.memory.availableMB‚ 1306.78125 disk.nvme0n1.in‚ 939.6640625 disk.nvme1n1.in‚ 744.57421875 disk.sda.in‚8.515625 disk.sdb1.out‚0 disk.md0p1.in‚ 12§v™y.296875# disk.nvme1n1.out‚45099.73828125 disk.sdb2.in‚3.78125$ memory_percent‚92.02404022216797 cpu‚25.194342742294868 proc.cpu.threads‚17 network.recv‚ 3.547483e+06! disk.sda.out‚1227.50732421875 proc.memory.rssMB‚3013.625* proc.memory.percent‚18.393707275390625 disk.sda1.out‚0 disk.sdb1.in‚0 network.sent‚ 2.4321083e+07$ disk./.usageGB‚2710.091075897217 disk.dm-0.in‚ 1684.23828125! disk.sdb.out‚1227.50732421875 disk.sda2.in‚8.515625 disk.sda1.in‚0 disk.md0p1.out‚ 1180.69140625! disk.dm-0.out‚160196.86328125" disk.sda2.out‚1227.50732421875694š'/ _wandbruntime‚11009  _runtime‚11009“ŽïHÇ›'Á _step‚761 & lr-AdamW/pg1‚0.0001617909169145224 & lr-AdamW/pg2‚0.0001617909169145224 trainer/global_step‚9299 $ _timestamp‚1.788506952188945e+09 _runtime‚11009.966129521ªQVÌœ'Æ & lr-AdamW/pg1‚0.0001617909169145224 & lr-AdamW/pg2‚0.0001617909169145224 trainer/global_step‚9299 $ _timestamp‚1.788506952188945e+09 _runtime‚11009.966129521 _step‚761ù é34'/ _wandbruntime‚11009  _runtime‚11009n Âú(ž'P D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 194323710 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10648037325312 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚793659362252800 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚793659362252800 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 235468133 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚863934033026560 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10608223090367 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56330323 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 194323708 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10648037325312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 429791843 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚653 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚651 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10648947296768 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 172759366 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚783010396953088 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚793658434278400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56330974 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11117875180544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚793658434278400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚767731599351535 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚9299 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 235441789ÊÊZ (Ÿ'†P YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 116428392 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 235441789 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 % _timestamp‚1.7885069521899967e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚875051932147200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 239267840 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10648037325312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 429791843 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚653 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚651 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10648947296768 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 172759366 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚783010396953088 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚793658434278400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56330974 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11117875180544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚793658434278400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚767731599351535 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚9299 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 235441789 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 194323710 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10648037325312 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚793659362252800 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚793659362252800 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 235468133 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚863934033026560 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10608223090367 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56330323 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 194323708 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 172776494 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚11010.904375777 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 235466620 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚875143322498048 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 116440457 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10649158785536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10649158785536 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚793741174824448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 429836860 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 235492990 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚757202410895072 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 429810488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚636 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 429836860 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚864024258731008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚767812659233929 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10609340363645 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56336673 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56336039 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 % _timestamp‚1.7885069531267517e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚783092034041856 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 194343868 trainer/global_step‚9299 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11118817349632 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10610248338857 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 194343870 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 172777130 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚757202393855712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚767811734219357 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 _step‚763 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 235492990 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚783092016038912 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10650068770304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚634 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segme C98nt.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370ÓM'¤'N ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56336039 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 % _timestamp‚1.7885069531267517e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚783092034041856 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 194343868 trainer/global_step‚9299 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11118817349632 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10610248338857 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 194343870 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 172777130 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚757202393855712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚767811734219357 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 _step‚763 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 235492990 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚783092016038912 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10650068770304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚634 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚793742102812160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 194343870 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 194343868 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 172776494 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚11010.904375777 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 235466620 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚875143322498048 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 116440457 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10649158785536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10649158785536 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚793741174824448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 429836860 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 235492990 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚757202410895072 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 429810488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚636 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 429836860 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚864024258731008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚767812659233929 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10609340363645 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56336673ûª4¥'/ _wandbruntime‚11010  _runtime‚11010<<ÄDa¦'Û 0loss/FRt1/assignment_loss‚1.9437824429535284 /loss/FRt4/assignment_loss‚2.391125676723627 1loss/SRqqt4/assignment_loss‚1.554782588373531 1loss/FBt4/assignment_loss‚0.11135104198526732 /loss/FRt1/detection_loss‚1.9567401445133246 epoch‚7 2loss/SRqqt1/assignment_loss‚1.0939108865956466 2loss/SRqqt2/assignment_loss‚1.2701477916152388 0loss/SRqqt2/detection_loss‚2.401346957242047 0loss/SRqqt4/detection_loss‚2.611092223362489 /loss/FRt3/detection_loss‚1.4089798938196438 0loss/SRqqt1/detection_loss‚2.496222995221615 0loss/SRqqt3/detection_loss‚0.983158046578164 _step‚764 0loss/FRt2/assignment_loss‚3.1228774285316465 0loss/FRt3/assignment_loss‚0.9917485172931965 2loss/SRqqt3/assignment_loss‚0.9376786978663418 1loss/FBt2/assignment_loss‚0.48513458597936093 /loss/FBt3/detection_loss‚2.0907257199287415 /loss/FRt2/detection_loss‚2.0583817851450292 /loss/FRt4/detection_loss‚2.9208998680114746 /loss/FBt1/detection_loss‚0.7116934196307109 &loss/total_loss‚37.878635273283685 _runtime‚11010.905928475 1loss/FBt1/assignment_loss‚0.09873918693822711 /loss/FBt4/detection_loss‚2.1655131618181866 3loss/FBt3/assignment_loss‚0.0034025318745989352 /loss/FBt2/detection_loss‚2.0691996812820435 trainer/global_step‚9299 % _timestamp‚1.7885069531289496e+09½õ7ìf§'à 0loss/FRt1/assignment_loss‚1.9437824429535284 /loss/FRt4/assignment_loss‚2.391125676723627 1loss/SRqqt4/assignment_loss‚1.554782588373531 1loss/FBt4/assignment_loss‚0.11135104198526732 /loss/FRt1/detection_loss‚1.9567401445133246 epoch‚7 2loss/SRqqt1/assignment_loss‚1.0939108865956466 2loss/SRqqt2/assignment_loss‚1.2701477916152388 0loss/SRqqt2/detection_loss‚2.401346957242047 0loss/SRqqt4/detection_loss‚2.611092223362489 /loss/FRt3/detection_loss‚1.4089798938196438 0loss/SRqqt1/detection_loss‚2.496222995221615 0loss/SRqqt3/detection_loss‚0.983158046578164 _step‚764 0loss/FRt2/assignment_loss‚3.1228774285316465 0loss/FRt3/assignment_loss‚0.9917485172931965 2loss/SRqqt3/assignment_loss‚0.9376786978663418 1loss/FBt2/assignment_loss‚0.48513458597936093 /loss/FBt3/detection_loss‚2.0907257199287415 /loss/FRt2/detection_loss‚2.0583817851450292 /loss/FRt4/detection_loss‚2.9208998680114746 /loss/FBt1/detection_loss‚0.7116934196307109 &loss/total_loss‚37.878635273283685 _runtime‚11010.905928475 1loss/FBt1/assignment_loss‚0.09873918693822711 /loss/FBt4/detection_loss‚2.1655131618181866 3loss/FBt3/assignment_loss‚0.0034025318745989352 /loss/FBt2/detection_loss‚2.0691996812820435 trainer/global_step‚9299 % _timestamp‚1.7885069531289496e+09ü?úᶓ¨'‚(:ˆ ×æéÔŸ¯­n gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚158.469' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.61725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|¸Íƒq©'‚(:æ ×æéÔƨçn* proc.memory.percent‚18.393707275390625 proc.cpu.threads‚17 disk.md0.in‚ 12.296875 disk.sdb2.in‚3.78125 cpu‚25.193835248052153 disk.sdb1.out‚0 disk.dm-0.in‚ 1684.23828125! disk.dm-0.out‚160565.30859375 disk.sda2.in‚8.515625# memory_percent‚92.0259952545166) disk./.usagePercent‚75.76202241253495 disk.sda1.out‚0! disk.sdb.out‚1230.85498046875 disk.md0.out‚ 1183.91796875" disk.sdb2.out‚1230.85498046875 proc.memory.rssMB‚3013.625! disk.sda.out‚1230.85498046875 disk.rbd0.out‚ 3.025390625 disk.nvme0n1.in‚ 939.6640625! disk.nvme0n1.out‚ 115385.90625# disk.nvme1n1.out‚45179.40234375 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.md0p1.out‚ 1183.91796875" disk.sda2.out‚1230.85498046875 network.recv‚ 3.557339e+06 disk.sdb.in‚3.78125 disk.rbd2.out‚ 3.38720703125( proc.memory.availableMB‚ 1306.4609375 network.sent‚ 2.4435949e+07% disk./.usageGB‚2709.6168937683105 disk.sda.in‚8.515625 disk.sda1.in‚0 disk.md0p1.in‚ 12.296875 disk.nvme1n1.in‚ 744.57421875©îìþ“ª'‚(:ˆ ææéԬؒm gpu.0.gpu‚36.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.219' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.55475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜyýÃn«'‚(:ã ææéÔáÓàm disk.sda.out‚1232.4013671875 disk.rbd0.out‚ 3.025390625 disk.sdb1.out‚0! disk.sda2.out‚1232.4013671875* proc.memory.percent‚18.393707275390625$ disk./.usageGB‚2709.566390991211 disk.md0.in‚ 12.296875 disk.sda1.in‚0) proc.memory.availableMB‚ 1306.46484375 disk.nvme0n1.in‚ 939.6640625 disk.nvme1n1.in‚ 744.57421875! disk.dm-0.out‚160693.56640625 network.sent‚ 2.4524674e+07$ disk.nvme0n1.out‚115435.40234375" disk.nvme1n1.out‚ 45258.1640625 disk.rbd2.in‚0 disk.rbd2.out‚ 3.38720703125 disk.dm-0.in‚ 1684.23828125 disk.sda2.in‚8.515625 disk.sda1.out‚0 disk.sda.in‚8.515625 disk.md0p1.out‚ 1185.39453125 disk.sdb.in‚3.78125 proc.memory.rssMB‚3013.625 disk.sdb2.in‚3.78125 disk.md0p1.in‚ 12.296875$ memory_percent‚92.02597141265869 cpu‚25.19355492953808 proc.cpu.threads‚17 disk.sdb.out‚1232.4013671875 disk.md0.out‚ 1185.39453125 disk.rbd0.in‚0 disk.sdb1.in‚0 network.recv‚ 3.570587e+06( disk./.usagePercent‚75.7606103337502! disk.sdb2.out‚1232.4013671875ˈ ]“¬'‚(:ˆ õæéÔ´­¹m gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚161.587' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.39675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´§p­'‚(:å õæéÔ­šn$ disk.nvme0n1.out‚115512.41015625! disk.nvme1n1.out‚ 45263.109375 disk.rbd2.out‚ 3.4111328125 proc.cpu.threads‚17 disk.sdb2.in‚3.78125 cpu‚25.19333588759415 disk.rbd2.in‚0 disk.sda1.in‚0 disk.rbd0.out‚ 3.025390625 disk.sdb1.out‚0 disk.md0p1.in‚ 12.296875! disk.sdb.out‚1233.87353515625 disk.nvme0n1.in‚ 939.6640625! disk.sda.out‚1233.87353515625$ disk./.usageGB‚2709.646266937256 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda.in‚8.515625 disk.sdb1.in‚0" disk.sdb2.out‚1233.87353515625 disk.md0.in‚ 12.296875 disk.md0.out‚ 1186.80859375 network.recv‚ 3.57401e+06) disk./.usagePercent‚75.76284369863228 disk.md0p1.out‚ 1186.80859375 disk.dm-0.in‚ 1684.23828125 disk.sda2.in‚8.515625$ memory_percent‚92.02597141265869 disk.nvme1n1.in‚ 744.57421875! disk.dm-0.out‚160775.51953125 disk.sdb.in‚3.78125" disk.sda2.out‚1233.87353515625) proc.memory.availableMB‚ 1306.46484375 proc.memory.rssMB‚3013.625* proc.memory.percent‚18.393707275390625 network.sent‚ 2.4532252e+07D%Á¯4®'/ _wandbruntime‚11067  _runtime‚11067 ‚l/ɯ'à ' lr-AdamW/pg1‚0.00015634933016244395 ' lr-AdamW/pg2‚0.00015634933016244395 trainer/global_step‚9349 $ _timestamp‚1.788507009970623e+09 _runtime‚11067.748089261 _step‚765åðµóΰ'È ' lr-AdamW/pg1‚0.00015634933016244395 ' lr-AdamW/pg2‚0.00015634933016244395 trainer/global_step‚9349 $ _timestamp‚1.788507009970623e+09 _runtime‚11067.748089261 _step‚765ýXw\F4±'/ _wandbruntime‚11067  _runtime‚11067T*V(²'P ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 195331708 _runtime‚11067.750305841 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 173652883 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚879621755649536 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚797796403969024 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 236713035 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚868445318248960 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚797795475994624 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚787091369247232 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚879621507121664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚787091369247232 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚761074273027728 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚787091351244288 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 117031640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚868445342189056 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10664100345475 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 195331708 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10704124750336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 173652223 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760oÝÔO³'†P RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10704124750336 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11176188872704 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 % _timestamp‚1.7885070099724267e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 236713035 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11176413460480 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚797796403969024 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10705034721792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 432044745 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚761074255988368 _step‚766 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚787091351244288 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 195331710 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 117031642 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 195331710 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56620583 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 173652883 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚879621755649536 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚797796403969024 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 236713035 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚868445318248960 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚797795475994624 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚787091369247232 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚879621507121664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚787091369247232 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚761074273027728 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚787091351244288 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 117031640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚868445342189056 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10664100345475 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 195331708 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10704124750336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 173652223 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 195331708 _runtime‚11067.750305841 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 432089762 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚868535543953408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 195351870 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚879713154565120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 195351868 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚787172970330112 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 195351870 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚771819416216237 % _timestamp‚1.7885070111802537e+09 _step‚767 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚761153273582912 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚879712899759104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 173670181 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10666125593965 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 195351868 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚771818491201665 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚797879144528384 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚797878216540672 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 236737892 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 117043705 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10705246210560 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 432089762 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚868535543953408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 195351870 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚879713154565120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 195351868 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚787172970330112 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 195351870 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚771819416216237 % _timestamp‚1.7885070111802537e+09 _step‚767 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚761153273582912 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚879712899759104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 173670181 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10666125593965 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 195351868 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚771818491201665 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚797879144528384 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚797878216540672 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 236737892 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 117043705 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10705246210560 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 237931593 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 174544739 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 196339710 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 196339708 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚791172305535488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚791172323538432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10720885575587 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 434271301 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚636 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚765025152754928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10761122146816 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚9399 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚770 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚872956603471360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚801933445685248 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56910483 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11234577824768 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚634 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚775745113316151 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 196339708 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 174545375 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 117634892 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚791172323538432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 237957937 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚884191427726848 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 237957937 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 237931593 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10760212175360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚791172305535488 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11234800315392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚801932517710848 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚801933445685248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10761122146816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56909849 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚801932517710848 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚801933445685248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10761122146816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56909849 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 237931593 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 174544739 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 196339710 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 196339708 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚791172305535488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚791172323538432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10720885575587 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 434271301 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚636 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚765025152754928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10761122146816 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚9399 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚770 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚872956603471360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚801933445685248 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56910483 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11234577824768 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚634 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚775745113316151 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 196339708 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 174545375 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 117634892 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚791172323538432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 237957937 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚884191427726848 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 237957937‚§õç”4È'/ _wandbruntime‚11129  _runtime‚11129ÑÁû®'#É'‹N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 434316292 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 196359868 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚884282822290432 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 237982794 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚642 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 237956424 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10762243620352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚802015258256896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10721094873861 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 174562484 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885070713482704e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10761333635584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚765104170349472 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56916169 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10762243620352 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚802016186244608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 196359868 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 174563126 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 196359870 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚873046853115904 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 237982794 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 434342664 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144,ÔìD'Ê'N T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚802016186244608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11235732017152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10761333635584 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 196359870 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 117646955 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56915529 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 434342664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚791253924621312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 117646957 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 434316292 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 196359868 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚884282822290432 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 237982794 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚642 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 237956424 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10762243620352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚802015258256896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10721094873861 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 174562484 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885070713482704e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10761333635584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚765104170349472 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56916169 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10762243620352 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚802016186244608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 196359868 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 174563126 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 196359870 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚873046853115904 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 237982794 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 434342664 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 175438616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚779752795312823 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 436524203 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚779751870298459 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10776762830695 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 % _timestamp‚1.7885071303159196e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10775854855691 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚795253259826688 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 239202839 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚768976032482128 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚795253259826688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10817209571840 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 197347710 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 239202839 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 175439261 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11293482705920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚806069559427072 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 436550549 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚888761395339776 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚768976015442768 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10816299600384 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 118238140 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 118238142 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 239176495 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10817209571840 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 197347710 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 197347708 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 175438616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚779752795312823 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 436524203 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚779751870298459 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10776762830695 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/reques™©ÂE ted_bytes.small_pool.peak‚ 1187728393 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 % _timestamp‚1.7885071303159196e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10775854855691 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚795253259826688 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 239202839 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚768976032482128 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚795253259826688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10817209571840 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 197347710 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 239202839 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 175439261 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11293482705920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚806069559427072 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 436550549 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚888761395339776 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚768976015442768 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10816299600384 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 118238140 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 118238142 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 239176495 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10817209571840 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 197347710 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 197347708 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 239201326 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚877558114398208 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚779832005166281 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚877558138338304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 trainer/global_step‚9449 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚645 % _timestamp‚1.7885071314247313e+09 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 118250205 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 239201326 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10777880104181 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 436595566 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚769055033037312 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 197367868 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 175456555 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚795334878912512 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚888852789966848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11294416568320 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚806153227960832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚888852530966528 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235060224 _step‚775 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10818331045376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚795334896915456 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚779832930180853 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚11189.202681438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 436595566 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10817421060608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10818331045376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 239201326 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚877558114398208 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚779832005166281 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚877558138338304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 trainer/global_step‚9449 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚645 % _timestamp‚1.7885071314247313e+09 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 118250205 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 239201326 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10777880104181 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 436595566 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚769055033037312 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 197367868 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 175456555 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚795334878912512 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚888852789966848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11294416568320 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚806153227960832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚888852530966528 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235060224 _step‚775 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10818331045376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚795334896915456 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚779832930180853 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚11189.202681438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 436595566 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10817421060608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10818331045376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚783758627280767 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 $ _timestamp‚1.788507190680759e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚799334214117888 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚893331036371456 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚799334232120832 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 438777105 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10873296996864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚881979197856256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚810206601143296 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 240447741 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57493493 _step‚778 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57492857 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10832640085803 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10872387025408 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 198355708 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10872387025408 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 198355710 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 198355710 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 trainer/global_step‚9499 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10873296996864 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 118841390 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚772926912209328 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 438803451 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚799334214117888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚881979173916160 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11351862455296 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144aCâ  (ï'…P D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚893331036371456 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚799334232120832 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 438777105 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10873296996864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚881979197856256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚810206601143296 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 240447741 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57493493 _step‚778 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57492857 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10832640085803 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10872387025408 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 198355708 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10872387025408 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 198355710 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 198355710 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 trainer/global_step‚9499 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10873296996864 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 118841390 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚772926912209328 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 438803451 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚799334214117888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚881979173916160 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11351862455296 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚638 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 176334247 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚783758627280767 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 $ _timestamp‚1.788507190680759e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚799334214117888 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0Š±4ð'/ _wandbruntime‚11249  _runtime‚11249Eó´¦ñ'‹N K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 198375868 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 438848468 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 176352113 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57498658 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚810290269677056 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 trainer/global_step‚9499 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10874418470400 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 176352740 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚882069423560704 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚779 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 118853457 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11353044787200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚783839687163161 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚627 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚783838762148589 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚799415851206656 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 240472598 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 240446228 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚799415833203712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10873508485632 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 240472598 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/alŒ´7k located_bytes.all.freed‚810289341689344 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10874418470400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57499283 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚773005912764512 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10832849384077 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 240446228 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885071917056065e+09Gc ¼'ò'N C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚783839687163161 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚627 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚783838762148589 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚799415851206656 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 240472598 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 240446228 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚799415833203712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10873508485632 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 240472598 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚810289341689344 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10874418470400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57499283 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚773005912764512 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10832849384077 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 240446228 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885071917056065e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 198375868 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 438848468 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 176352113 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57498658 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚810290269677056 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 trainer/global_step‚9499 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10874418470400 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 176352740 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚882069423560704 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚779 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 118853457 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11353044787200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0‹áƒ˜4ó'/ _wandbruntime‚11249  _runtime‚11249Þ.ë—aô'Û 0loss/FRt3/assignment_loss‚1.1832447262910697 2loss/SRqqt1/assignment_loss‚1.6972432603325807 2loss/SRqqt2/assignment_loss‚2.0318703204393387 0loss/SRqqt2/detection_loss‚2.126781557287489 /loss/FBt3/detection_loss‚1.7904868125915527 _runtime‚11249.485631368 /loss/FRt2/detection_loss‚2.3871568880975245 /loss/FRt3/detection_loss‚1.7087202857320125 1loss/SRqqt1/detection_loss‚1.9144253396626674 % _timestamp‚1.7885071917086263e+09 _step‚780 2loss/FBt3/assignment_loss‚0.003008212818940592 /loss/FRt1/detection_loss‚1.6153794134635246 0loss/FBt2/assignment_loss‚0.0979629485508566 0loss/FRt2/assignment_loss‚3.3753899067640303 2loss/SRqqt4/assignment_loss‚1.2508920718516623 1loss/FBt4/assignment_loss‚0.40569602793223114 /loss/FRt4/detection_loss‚3.0520693328645496 1loss/SRqqt4/detection_loss‚2.5788358620234897 &loss/total_loss‚40.612277517226076 trainer/global_step‚9499 /loss/FRt4/assignment_loss‚3.049152202076382 1loss/SRqqt3/detection_loss‚0.7133671287995051 /loss/FBt2/detection_loss‚3.4237211412853665 epoch‚7 /loss/FRt1/assignment_loss‚2.396151939630508 2loss/SRqqt3/assignment_loss‚1.0060348146858706 1loss/FBt1/assignment_loss‚0.09582307416217262 .loss/FBt1/detection_loss‚0.600353104469832 /loss/FBt4/detection_loss‚2.1085111454129217Ï}Tðfõ'à 0loss/FRt2/assignment_loss‚3.3753899067640303 2loss/SRqqt4/assignment_loss‚1.2508920718516623 1loss/FBt4/assignment_loss‚0.40569602793223114 /loss/FRt4/detection_loss‚3.0520693328645496 1loss/SRqqt4/detection_loss‚2.5788358620234897 &loss/total_loss‚40.612277517226076 trainer/global_step‚9499 /loss/FRt4/assignment_loss‚3.049152202076382 1loss/SRqqt3/detection_loss‚0.7133671287995051 /loss/FBt2/detection_loss‚3.4237211412853665 epoch‚7 /loss/FRt1/assignment_loss‚2.396151939630508 2loss/SRqqt3/assignment_loss‚1.0060348146858706 1loss/FBt1/assignment_loss‚0.09582307416217262 .loss/FBt1/detection_loss‚0.600353104469832 /loss/FBt4/detection_loss‚2.1085111454129217 0loss/FRt3/assignment_loss‚1.1832447262910697 2loss/SRqqt1/assignment_loss‚1.6972432603325807 2loss/SRqqt2/assignment_loss‚2.0318703204393387 0loss/SRqqt2/detection_loss‚2.126781557287489 /loss/FBt3/detection_loss‚1.7904868125915527 _runtime‚11249.485631368 /loss/FRt2/detection_loss‚2.3871568880975245 /loss/FRt3/detection_loss‚1.7087202857320125 1loss/SRqqt1/detection_loss‚1.9144253396626674 % _timestamp‚1.7885071917086263e+09 _step‚780 2loss/FBt3/assignment_loss‚0.003008212818940592 /loss/FRt1/detection_loss‚1.6153794134635246 0loss/FBt2/assignment_loss‚0.0979629485508566Œû­Š‘ö'‚(:† ¸èéÔ˜ï—m gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.09' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.7725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E_ñ°a÷'‚(:Ö ¸èéÔú™àn disk.sdb2.out‚ 1256.17578125 disk.sda1.out‚0 disk.md0p1.out‚ 1208.24609375 disk.nvme1n1.in‚ 744.57421875 disk.sdb2.in‚3.78125 disk.rbd0.in‚0 disk.md0p1.in‚ 12.296875 disk.sdb.out‚ 1256.17578125 disk.nvme0n1.in‚ 939.6640625) disk./.usagePercent‚75.80485610896918 disk.rbd2.in‚0 disk.dm-0.in‚ 1684.23828125! disk.dm-0.out‚163471.38671875 disk.sdb.in‚3.78125 disk.md0.out‚ 1208.24609375 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.sda2.out‚ 1256.17578125 disk.nvme0n1.out‚ 116113.625# memory_percent‚92.0274019241333 cpu‚25.19047748888688 network.sent‚ 2.4941181e+07 network.recv‚ 3.666806e+06% disk./.usageGB‚2711.1488342285156 disk.sda.in‚8.515625 disk.sdb1.in‚0 disk.sda.out‚ 1256.17578125 disk.sda1.in‚0 disk.rbd0.out‚ 3.025390625 proc.memory.rssMB‚3013.625 disk.rbd2.out‚ 3.537109375 disk.md0.in‚ 12.296875* proc.memory.percent‚18.393707275390625 disk.sda2.in‚8.515625# disk.nvme1n1.out‚47357.76171875) proc.memory.availableMB‚ 1306.23046875p<o“ø'‚(:ˆ ÇèéÔ“·úl gpu.0.gpu‚37.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.825' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.70625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%ó¬ueù'‚(:Ú ÇèéÔð¸¯m disk.rbd0.in‚0 disk.rbd0.out‚ 3.025390625! disk.sda2.out‚1257.5263671875 disk.nvme1n1.in‚ 744.57421875* proc.memory.percent‚18.393707275390625 disk.nvme0n1.out‚ 116113.625' disk./.usagePercent‚75.760538337891! disk.sdb2.out‚1257.5263671875 disk.sda.out‚1257.5263671875 disk.sdb.out‚1257.5263671875 disk.sda2.in‚8.515625 disk.md0.in‚ 12.296875 disk.nvme0n1.in‚ 939.6640625% disk./.usageGB‚2709.5638160705566 disk.rbd2.in‚0 disk.sdb2.in‚3.78125 disk.sda.in‚8.515625 disk.sdb1.out‚0 disk.sdb.in‚3.78125 network.recv‚ 3.672115e+06 disk.rbd2.out‚ 3.537109375 disk.sda1.in‚0 disk.sdb1.in‚0 disk.dm-0.out‚163471.6015625$ memory_percent‚92.02773571014404) proc.memory.availableMB‚ 1306.17578125 cpu‚25.190252998855904 disk.md0p1.in‚ 12.296875 proc.memory.rssMB‚3013.625 proc.cpu.threads‚17 network.sent‚ 2.5037958e+07 disk.sda1.out‚0 disk.md0p1.out‚ 1209.5390625 disk.dm-0.in‚ 1684.23828125" disk.nvme1n1.out‚ 47357.9765625 disk.md0.out‚ 1209.5390625~8úaú'‚(:Ö ÖèéÔ’”Ål proc.memory.rssMB‚3013.625# disk.nvme1n1.out‚47359.22265625$ proc.memory.availableMB‚1306.125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.md0p1.out‚ 1211.25390625) disk./.usagePercent‚75.75913660517787 disk.sdb.out‚ 1259.29296875 disk.rbd2.out‚ 3.537109375# disk.nvme0n1.out‚116114.0390625 disk.md0.out‚ 1211.25390625! disk.dm-0.out‚163473.26171875$ disk./.usageGB‚2709.513683319092 disk.rbd2.in‚0 disk.sda1.out‚0 disk.rbd0.out‚ 3.025390625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚ 12.296875 network.recv‚ 3.676975e+06 disk.rbd0.in‚0 disk.dm-0.in‚ 1684.23828125$ memory_percent‚92.02804565429688 disk.md0.in‚ 12.296875 network.sent‚ 2.5044337e+07 disk.sdb.in‚3.78125 disk.sda2.in‚8.515625 disk.nvme0n1.in‚ 939.6640625 disk.nvme1n1.in‚ 744.57421875 disk.sdb2.in‚3.78125 disk.sdb2.out‚ 1259.29296875 disk.sda2.out‚ 1259.29296875 disk.sda.in‚8.515625 disk.sda.out‚ 1259.29296875* proc.memory.percent‚18.393707275390625 cpu‚25.190048681387815–›Þ‘û'‚(:† ÖèéÔ€«âm gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚171.944' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.986 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_¾ÿ9“ü'‚(:ˆ åèéÔƒ¹¸m gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.383' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.09575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õä=pý'‚(:å åèéÔ®‰óm disk.rbd2.out‚ 3.537109375! disk.sda.out‚1260.97900390625 disk.sda1.out‚0" disk.sda2.out‚1260.97900390625 cpu‚25.189812619023837 network.sent‚ 2.5049625e+07 disk.rbd2.in‚0 disk.sdb1.in‚0 disk.sdb2.in‚3.78125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.md0p1.in‚ 12.296875* proc.memory.percent‚18.393707275390625 disk.nvme1n1.in‚ 744.57421875 disk.dm-0.in‚ 1684.23828125 disk.sda.in‚8.515625 disk.sdb.in‚3.78125# disk.nvme0n1.out‚116114.0390625% disk./.usageGB‚2709.5907669067383 disk.md0.out‚ 1212.8828125" disk.sdb2.out‚1260.97900390625! disk.dm-0.out‚163646.94921875 proc.cpu.threads‚17 network.recv‚ 3.680491e+06 disk.sdb1.out‚0 disk.nvme0n1.in‚ 939.6640625( proc.memory.availableMB‚ 1306.1015625) disk./.usagePercent‚75.76129189455045# disk.nvme1n1.out‚47532.91015625 disk.rbd0.out‚ 3.025390625 disk.md0p1.out‚ 1212.8828125! disk.sdb.out‚1260.97900390625 disk.sda2.in‚8.515625$ memory_percent‚92.02818870544434 proc.memory.rssMB‚3013.625 disk.md0.in‚ 12.296875ÊJnJ4þ'/ _wandbruntime‚11300  _runtime‚11300ë¸è%Êÿ'Ä % _timestamp‚1.7885072429051144e+09 _runtime‚11300.682679063 _step‚781 ' lr-AdamW/pg1‚0.00013536479606618006 ' lr-AdamW/pg2‚0.00013536479606618006 trainer/global_step‚9549Úތπ(É ' lr-AdamW/pg1‚0.00013536479606618006 ' lr-AdamW/pg2‚0.00013536479606618006 trainer/global_step‚9549 % _timestamp‚1.7885072429051144e+09 _runtime‚11300.682679063 _step‚781—‡4(/ _wandbruntime‚11300  _runtime‚11300Snwß(‚(P M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚897901144863232 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 441030007 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10928474450432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 177229989 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 119444640 _runtime‚11300.683701888 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 199363708 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚619 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11410661784576 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10887609365907 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 177229368 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 _step‚782 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 241666299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11410443488256 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10888517340911 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 % _timestamp‚1.7885072429063418e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚803415186412032 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 241666299 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚814343642859520 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 199363710 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57785347 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚776877791936528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10928474450432 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 441056353 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚787766309277439 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10929384421888 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚814343642859520 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 119444642 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚897900902626816 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 241692643 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚803415186412032 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10929384421888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 241692643 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242236416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚886490459138560 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615”A˜­ƒ(†P G?DeviceStatsMonÚ¨Tòô'itor.on_train_batch_start/segment.large_pool.peak‚615 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚803415186412032 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 241666299 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚814343642859520 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 199363710 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57785347 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚776877791936528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10928474450432 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 441056353 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚787766309277439 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10929384421888 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚814343642859520 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 119444642 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚897900902626816 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 241692643 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚803415186412032 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10929384421888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 241692643 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242236416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚886490459138560 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚897901144863232 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 441030007 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10928474450432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 177229989 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 119444640 _runtime‚11300.683701888 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 199363708 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚619 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11410661784576 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10887609365907 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 177229368 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 _step‚782 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 241666299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11410443488256 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10888517340911 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 % _timestamp‚1.7885072429063418e+09 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚11301.704240679 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚814427311393280 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 441074998 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 241691130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚803496805497856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 trainer/global_step‚9549 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 241717500 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10888726639185 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 441101370 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 119456707 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚622 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 199383868 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚803496805497856 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11411834049536 _step‚783 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚897992288026624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 177247328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚886580684843008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚814426383405568 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚897992542832640 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10889634614397 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 199383868 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 199383870ÌFß½'†(N MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚897992288026624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 177247328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚886580684843008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚814426383405568 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚897992542832640 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10889634614397 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 199383868 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 199383870 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7885072439259617e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10929595910656 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 177247952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 241717500 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 199383870 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚814427311393280 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚11301.704240679 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚814427311393280 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 441074998 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 241691130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚803496805497856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 trainer/global_step‚9549 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 241717500 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10888726639185 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 441101370 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 119456707 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚622 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 199383868 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚803496805497856 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11411834049536 _step‚7836Ø64‡(/ _wandbruntime‚11301  _runtime‚11301!>$>[ˆ(Õ 2loss/SRqqt1/assignment_loss‚1.2829683322572347 1loss/FBt3/assignment_loss‚0.10142356802725772 .loss/FBt1/detection_loss‚0.761075655570956 .loss/FBt2/detection_loss‚2.006830362712636 /loss/FBt4/detection_loss‚1.9031555950641632 /loss/FRt4/assignment_loss‚2.934912152150098 1loss/FBt2/assignment_loss‚0.32434168138618535 /loss/FRt4/detection_loss‚2.6840977177900425 1loss/FBt4/assignment_loss‚0.20357024055443276 /loss/FRt1/detection_loss‚1.6914805615084163 0loss/SRqqt1/detection_loss‚2.144993675477577 trainer/global_step‚9549 _runtime‚11301.705323124 /loss/FRt2/detection_loss‚2.3400497291503206 0loss/SRqqt2/detection_loss‚2.727534480392933 /loss/FBt3/detection_loss‚2.3175494770208993 &loss/total_loss‚39.325864869228795 0loss/FRt1/assignment_loss‚1.9886565180711968 0loss/FRt2/assignment_loss‚3.1525109977949235 2loss/SRqqt2/assignment_loss‚1.7212773516774178 1loss/FBt1/assignment_loss‚0.15104277211281902 0loss/SRqqt4/detection_loss‚2.669052216139707 /loss/FRt3/assignment_loss‚1.350623963510289 1loss/SRqqt3/assignment_loss‚0.985131290235896 1loss/SRqqt4/assignment_loss‚1.333034566857598 0loss/SRqqt3/detection_loss‚0.877151493271885 % _timestamp‚1.7885072439282832e+09 _step‚784 /loss/FRt3/detection_loss‚1.6734004704939092 epoch‚7«¶ögñ‰(Ú 1loss/FBt4/assignment_loss‚0.20357024055443276 /loss/FRt1/detection_loss‚1.6914805615084163 0loss/SRqqt1/detection_loss‚2.144993675477577 trainer/global_step‚9549 _runtime‚11301.705323124 /loss/FRt2/detection_loss‚2.3400497291503206 0loss/SRqqt2/detection_loss‚2.727534480392933 /loss/FBt3/detection_loss‚2.3175494770208993 &loss/total_loss‚39.325864869228795 0loss/FRt1/assignment_loss‚1.9886565180711968 0loss/FRt2/assignment_loss‚3.1525109977949235 2loss/SRqqt2/assignment_loss‚1.7212773516774178 1loss/FBt1/assignment_loss‚0.15104277211281902 0loss/SRqqt4/detection_loss‚2.669052216139707 /loss/FRt3/assignment_loss‚1.350623963510289 1loss/SRqqt3/assignment_loss‚0.985131290235896 1loss/SRqqt4/assignment_loss‚1.333034566857598 0loss/SRqqt3/detection_loss‚0.877151493271885 % _timestamp‚1.7885072439282832e+09 _step‚784 /loss/FRt3/detection_loss‚1.6734004704939092 epoch‚7 2loss/SRqqt1/assignment_loss‚13Ë lo.2829683322572347 1loss/FBt3/assignment_loss‚0.10142356802725772 .loss/FBt1/detection_loss‚0.761075655570956 .loss/FBt2/detection_loss‚2.006830362712636 /loss/FBt4/detection_loss‚1.9031555950641632 /loss/FRt4/assignment_loss‚2.934912152150098 1loss/FBt2/assignment_loss‚0.32434168138618535 /loss/FRt4/detection_loss‚2.6840977177900425z€ß`‘Š(‚(:† ôèéÔü’Êk gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.916' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.229 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘g(i‹(‚(:Þ ôèéÔÕͱm disk.sdb.out‚1262.3193359375! disk.sda2.out‚1262.3193359375 disk.nvme1n1.in‚ 744.57421875! disk.nvme1n1.out‚ 47593.421875 disk.md0.out‚ 1214.16015625* proc.memory.percent‚18.393707275390625 network.sent‚ 2.5053424e+07 network.recv‚ 3.685008e+06 disk.sdb.in‚3.78125 disk.sda.in‚8.515625 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.18956386576666 disk.sdb1.out‚0 disk.dm-0.out‚163707.7109375 disk.nvme0n1.in‚ 939.6640625 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.md0p1.in‚ 12.296875# disk.nvme0n1.out‚116114.2890625 disk.sdb2.in‚3.78125' proc.memory.availableMB‚ 1305.828125% disk./.usageGB‚2709.7159118652344 disk.md0p1.out‚ 1214.16015625 disk.sda2.in‚8.515625) disk./.usagePercent‚75.76479099996723 disk.sdb1.in‚0 disk.rbd2.out‚ 3.537109375! disk.sdb2.out‚1262.3193359375 disk.sda.out‚1262.3193359375$ memory_percent‚92.02985763549805 disk.rbd0.out‚ 3.025390625 disk.dm-0.in‚ 1684.23828125 disk.md0.in‚ 12.296875 proc.memory.rssMB‚3013.625 proc.cpu.threads‚17 ðWèoŒ(‚(:ä ƒééÔDzël" disk.sdb2.out‚1264.51123046875" disk.sda2.out‚1264.51123046875! disk.nvme0n1.out‚ 116114.40625 disk.rbd2.in‚0 disk.rbd2.out‚ 3.537109375 cpu‚25.189337880883667 network.sent‚ 2.516311e+07% disk./.usageGB‚2709.7416343688965* proc.memory.percent‚18.393707275390625 network.recv‚ 3.690421e+06) disk./.usagePercent‚75.76551021193532 disk.sda1.out‚0 disk.sdb1.in‚0 disk.sda2.in‚8.515625 disk.sdb2.in‚3.78125 disk.rbd0.in‚0 disk.md0p1.out‚ 1216.26953125 disk.nvme0n1.in‚ 939.6640625 proc.memory.rssMB‚3013.625 proc.cpu.threads‚17 disk.sda.in‚8.515625 disk.sda1.in‚0 disk.md0.in‚ 12.296875 disk.rbd0.out‚ 3.025390625# disk.nvme1n1.out‚47739.01171875 disk.md0.out‚ 1216.26953125( proc.memory.availableMB‚ 1306.2578125 disk.sdb1.out‚0 disk.dm-0.in‚ 1684.23828125 disk.sdb.in‚3.78125! disk.sdb.out‚1264.51123046875 disk.nvme1n1.in‚ 744.57421875$ memory_percent‚92.02723503112793! disk.sda.out‚1264.51123046875 disk.md0p1.in‚ 12.296875! disk.dm-0.out‚163853.41796875²”€Å“(‚(:ˆ ƒééÔ€ÍÍm gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚164.039' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.00975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹ n¥“Ž(‚(:ˆ ’ééÔø°Ák gpu.0.gpu‚49.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜ö)l(‚(:á ’ééÔ‰ÛÙl network.sent‚ 2.5167227e+07 disk.md0p1.in‚ 12.296875$ memory_percent‚92.03181266784668 disk.md0.in‚ 12.296875 disk.rbd2.out‚ 3.537109375 disk.sda1.in‚0! disk.sdb.out‚1265.74853515625( proc.memory.availableMB‚ 1305.5078125 proc.memory.rssMB‚3013.625 network.recv‚ 3.695201e+06 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.dm-0.in‚ 1684.23828125 disk.dm-0.out‚164046.1171875" disk.sda2.out‚1265.74853515625 disk.sdb2.in‚3.78125! disk.sda.out‚1265.74853515625 disk.rbd0.out‚ 3.025390625 disk.md0p1.out‚ 1217.453125 cpu‚25.18906664143166% disk./.usageGB‚2710.1445274353027 disk.md0.out‚ 1217.453125 disk.nvme0n1.in‚ 939.6640625$ disk.nvme0n1.out‚116115.58203125 disk.sda.in‚8.515625 disk.sda1.out‚0 disk.sdb.in‚3.78125* proc.memory.percent‚18.393707275390625 disk.nvme1n1.in‚ 744.57421875" disk.sdb2.out‚1265.74853515625 disk.sdb1.out‚0( disk./.usagePercent‚75.7767752706959# disk.nvme1n1.out‚47930.53515625 disk.rbd2.in‚0 disk.sda2.in‚8.515625 proc.cpu.threads‚17±«N“(‚(:ˆ ¡ééÔü“ïk gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.219' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.55475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 áé·l‘(‚(:á ¡ééÔ˜ÌÔp) disk./.usagePercent‚75.77689793030784 disk.nvme0n1.in‚ 939.6640625 disk.sda.in‚8.515625 disk.rbd0.out‚ 3.025390625! disk.sdb.out‚1267.18505859375 proc.cpu.threads‚17 disk.sdb.in‚3.78125 network.sent‚ 2.5174694e+07 disk.rbd0.in‚0 disk.md0p1.in‚ 12.296875 disk.sda2.in‚8.515625" disk.sda2.out‚1267.18505859375 proc.memory.rssMB‚3013.625$ disk.nvme0n1.out‚116115.58984375 disk.nvme1n1.in‚ 744.57421875 disk.sda1.in‚0 disk.sda1.out‚0" disk.sdb2.out‚1267.18505859375 disk.rbd2.in‚0 disk.md0.out‚ 1218.82421875 disk.sdb1.out‚0 disk.md0p1.out‚ 1218.82421875 cpu‚25.188821037635357$ disk./.usageGB‚2710.148914337158 disk.rbd2.out‚ 3.537109375 disk.sdb2.in‚3.78125 disk.sdb1.in‚0 disk.dm-0.out‚164268.7109375$ memory_percent‚92.03033447265625 disk.dm-0.in‚ 1684.23828125# proc.memory.availableMB‚1305.75* proc.memory.percent‚18.393707275390625 network.recv‚ 3.698415e+06# disk.nvme1n1.out‚48153.12109375 disk.md0.in‚ 12.296875! disk.sda.out‚1267.18505859375·äaT4’(/ _wandbruntime‚11355  _runtime‚11355KCÙÊ“(Ä _step‚785 ' lr-AdamW/pg1‚0.00013031958231587482 ' lr-AdamW/pg2‚0.00013031958231587482 trainer/global_step‚9599 % _timestamp‚1.7885072972994828e+09 _runtime‚11355.076711624¯s½ZÏ”(É % _timestamp‚1.7885072972994828e+09 _runtime‚11355.076711624 _step‚785 ' lr-AdamW/pg1‚0.00013031958231587482 ' lr-AdamW/pg2‚0.00013031958231587482 trainer/global_step‚9599‘4JO4•(/ _wandbruntime‚11355  _runtime‚11355iâý²(–(P [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10984561875456 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 200371710 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚791772141245383 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10944394596019 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 trainer/global_step‚9599 % _timestamp‚1.7885072973007686e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10985471846912 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _step‚786 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚807496140703232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚891001744360960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 200371710 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚818480684575744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 443309255 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚902470751451136 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚818480684575744 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11469231677952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚639 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 120047890 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚818481612550144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _runtime‚11355.078135051 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚807496122700288 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 242937545 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 178124698 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚807496140703232 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 242937545 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 178124059 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚891001768301056 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11469007090176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 242911201 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10985471846912 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚902470999979008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚780828654624368 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 200371708 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725íõ^f (—(†P ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚807496140703232 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 242937545 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 178124059 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚891001768301056 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11469007090176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 242911201 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10985471846912 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚902470999979008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚780828654624368 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 200371708 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10984561875456 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 200371710 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚791772141245383 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10944394596019 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 trainer/global_step‚9599 % _timestamp‚1.7885072973007686e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10985471846912 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _step‚786 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚807496140703232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚891001744360960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 200371710 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚818480684575744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 443309255 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚902470751451136 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚818480684575744 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11469231677952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚639 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 120047890 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚818481612550144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _runtime‚11355.078135051 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚807496122700288 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 242937545 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 178124698 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144’TU84˜(/ _wandbruntime‚11356  _runtime‚11356_àØ™(‹N E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚791853201127777 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚791852276113205 % _timestamp‚1.7885072984149523e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 443327900 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 _step‚787 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10945511869505 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 178141870 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11470397536768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 443327900 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 200391868 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚807577759789056 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 242936032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 120059955 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚891091970065408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚807577741786112 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚902562134639104 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 200391870 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 443354272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11470164573696 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=Dž¨ 9eviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 120059957 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 trainer/global_step‚9599 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚807577741786112 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10985683335680 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 178142509 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 200391868 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 _runtime‚11356.194405708 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58081915 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚902562391542272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10944603894293 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 443354272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11470164573696 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 120059957 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 trainer/global_step‚9599 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚807577741786112 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10985683335680 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 178142509 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 200391868 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 _runtime‚11356.194405708 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58081915 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚902562391542272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10944603894293 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 443327900 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 _step‚787 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10945511869505 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 178141870 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11470397536768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0“ G ›4›(/ _wandbruntime‚11356  _runtime‚11356ËžT\œ(Ö /loss/FBt1/detection_loss‚1.0858818512704027 &loss/total_loss‚43.064122232782054 /loss/FRt4/detection_loss‚2.9236991902192435 _runtime‚11356.195824004 0loss/FRt4/assignment_loss‚3.0487583213382297 0loss/FRt3/assignment_loss‚1.6881364113847974 .loss/FBt4/detection_loss‚2.696061056852341 _step‚788 0loss/SRqqt1/detection_loss‚2.061953309281119 epoch‚7 2loss/SRqqt4/assignment_loss‚1.8794776083066547 /loss/FRt3/detection_loss‚1.5994999310605122 2loss/SRqqt2/assignment_loss‚1.4612380602594577 1loss/FBt1/assignment_loss‚0.08448213161020235 1loss/SRqqt2/detection_loss‚2.3493332465489707 1loss/SRqqt4/detection_loss‚2.4835150262405135 /loss/FBt3/detection_loss‚2.8680368761221566 % _timestamp‚1.7885072984185941e+09 1loss/SRqqt3/detection_loss‚0.7363738789484441 .loss/FBt2/detection_loss‚2.425504025660063 trainer/global_step‚9599 1loss/FBt2/assignment_loss‚0.13290917679052272 /loss/FRt1/assignment_loss‚2.699721654256185 /loss/FRt2/assignment_loss‚3.024198020205778 2loss/SRqqt1/assignment_loss‚1.3128652646495351 2loss/SRqqt3/assignment_loss‚0.7952572411736942 .loss/FBt3/assignment_loss‚0.63581290532602 1loss/FBt4/assignment_loss‚0.13389856619672477 /loss/FRt1/detection_loss‚2.3126293175408383 .loss/FRt2/detection_loss‚2.624879161539652®ðTEa(Û _runtime‚11356.195824004 0loss/FRt4/assignment_loss‚3.0487583213382297 0loss/FRt3/assignment_loss‚1.6881364113847974 .loss/FBt4/detection_loss‚2.696061056852341 _step‚788 0loss/SRqqt1/detection_loss‚2.061953309281119 epoch‚7 2loss/SRqqt4/assignment_loss‚1.8794776083066547 /loss/FRt3/detection_loss‚1.5994999310605122 2loss/SRqqt2/assignment_loss‚1.4612380602594577 1loss/FBt1/assignment_loss‚0.08448213161020235 1loss/SRqqt2/detection_loss‚2.3493332465489707 1loss/SRqqt4/detection_loss‚2.4835150262405135 /loss/FBt3/detection_loss‚2.8680368761221566 % _timestamp‚1.7885072984185941e+09 1loss/SRqqt3/detection_loss‚0.7363738789484441 .loss/FBt2/detection_loss‚2.425504025660063 trainer/global_step‚9599 1loss/FBt2/assignment_loss‚0.13290917679052272 /loss/FRt1/assignment_loss‚2.699721654256185 /loss/FRt2/assignment_loss‚3.024198020205778 2loss/SRqqt1/assignment_loss‚1.3128652646495351 2loss/SRqqt3/assignment_loss‚0.7952572411736942 .loss/FBt3/assignment_loss‚0.63581290532602 1loss/FBt4/assignment_loss‚0.13389856619672477 /loss/FRt1/detection_loss‚2.3126293175408383 .loss/FRt2/detection_loss‚2.624879161539652 /loss/FBt1/detection_loss‚1.0858818512704027 &loss/total_loss‚43.064122232782054 /loss/FRt4/detection_loss‚2.9236991902192435”B|Ç"ož(‚(:ä °ééÔ˾±m disk.rbd0.out‚ 3.025390625 disk.dm-0.in‚ 1684.23828125 disk.sdb.in‚3.78125 cpu‚25.188609570862337 disk.nvme1n1.in‚ 744.57421875 disk.md0.in‚ 12.296875 disk.nvme0n1.in‚ 939.6640625$ disk.nvme0n1.out‚116117.25390625 disk.rbd2.in‚0 disk.rbd2.out‚ 3.537109375 disk.sda.in‚8.515625! disk.sda.out‚1268.75732421875) disk./.usagePercent‚75.77689793030784 disk.sdb2.in‚3.78125 disk.sdb1.in‚0" disk.sda2.out‚1268.75732421875) proc.memory.availableMB‚ 1305.51953125 proc.cpu.threads‚17 network.sent‚ 2.5178427e+07$ disk./.usageGB‚2710.148914337158" disk.sdb2.out‚1268.75732421875 disk.sdb1.out‚0! disk.sdb.out‚1268.75732421875 disk.sda2.in‚8.515625$ memory_percent‚92.03174114227295# disk.nvme1n1.out‚48244.73046875 disk.md0.out‚ 1220.3359375 disk.sda1.in‚0 disk.md0p1.in‚ 12.296875 proc.memory.rssMB‚3013.625* proc.memory.percent‚18.393707275390625 disk.rbd0.in‚0 disk.md0p1.out‚ 1220.3359375 disk.dm-0.out‚ 164361.984375 network.recv‚ 3.703032e+06 disk.sda1.out‚0ӊГ’Ÿ(‚(:‡ °ééÔ½Çâm gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚179.246' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.8115 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕN@Žo (‚(:ä ¿ééÔÚ¨Îl disk.md0p1.in‚ 12.296875 cpu‚25.188387194984525# disk.nvme0n1.out‚116117.2578125 disk.md0.out‚ 1222.140625 disk.sda.in‚8.515625 proc.cpu.threads‚17) disk./.usagePercent‚75.76564684407698 disk.nvme0n1.in‚ 939.6640625 disk.sdb2.in‚3.78125 disk.rbd0.in‚0 disk.rbd0.out‚ 3.025390625 disk.sdb1.in‚0 disk.sdb.in‚3.78125 disk.md0.in‚ 12.296875! disk.sda.out‚1270.61669921875$ memory_percent‚92.02873706817627* proc.memory.percent‚18.393707275390625 network.recv‚ 3.707896e+06" disk.sdb2.out‚1270.61669921875) proc.memory.availableMB‚ 1306.01171875 proc.memory.rssMB‚3013.625" disk.sda2.out‚1270.61669921875 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sda2.in‚8.515625 disk.md0p1.out‚ 1222.140625! disk.dm-0.out‚164472.87109375 disk.rbd2.out‚ 3.537109375 network.sent‚ 2.5285238e+07% disk./.usageGB‚2709.7465209960938# disk.nvme1n1.out‚48355.61328125 disk.sdb1.out‚0 disk.dm-0.in‚ 1684.23828125! disk.sdb.out‚1270.61669921875 disk.nvme1n1.in‚ 744.57421875 disk.sda1.in‚03» {’¡(‚(:‡ ¿ééÔˆ˜m gpu.0.gpu‚35.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚145.438' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.3595 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ý⇑¢(‚(:† ÎééÔÊœ¯m gpu.0.gpu‚44.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚124.044' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.011 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ嵩j£(‚(:ß ÎééÔ³­ám disk.rbd2.out‚ 3.537109375" disk.sdb2.out‚1272.24169921875 disk.md0.out‚ 1223.703125! disk.sda.out‚1272.24169921875 cpu‚25.188200132728937 disk.md0p1.in‚ 12.296875 disk.nvme0n1.in‚ 939.6640625# memory_percent‚92.0278787612915 proc.memory.rssMB‚3013.625 disk.dm-0.in‚ 1684.23828125 disk.dm-0.out‚ 164593.875 network.sent‚ 2.5291525e+07% disk./.usageGB‚2710.0723991394043 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0 disk.nvme1n1.in‚ 744.57421875 disk.md0.in‚ 12.296875) disk./.usagePercent‚75.77475853335454 disk.sda1.out‚0 disk.rbd0.out‚ 3.025390625 disk.md0p1.out‚ 1223.703125! disk.sdb.out‚1272.24169921875# disk.nvme1n1.out‚48473.56640625 disk.rbd2.in‚0 network.recv‚ 3.712947e+06 disk.sdb1.in‚0 disk.sdb.in‚3.78125 disk.sda2.in‚8.515625" disk.sda2.out‚1272.24169921875* proc.memory.percent‚18.393707275390625 disk.sdb2.in‚3.78125 disk.sda.in‚8.515625 proc.cpu.threads‚17$ disk.nvme0n1.out‚116120.30859375) proc.memory.availableMB‚ 1306.15234375ðw_r’¤(‚(:‡ ÝééÔÂþ¦k gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚134.298' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.5745 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡+Íq¥(‚(:æ ÝééÔ¿ØÀm$ disk./.usageGB‚2710.148937225342 disk.sda1.out‚0 disk.sdb.in‚3.78125 disk.sda2.in‚8.515625* proc.memory.percent‚18.393707275390625 cpu‚25.187976401423473 disk.rbd0.out‚ 3.025390625 disk.nvme0n1.in‚ 939.6640625$ disk.nvme0n1.out‚116120.30859375! disk.nvme1n1.out‚ 48722.703125 disk.md0.in‚ 12.296875$ memory_percent‚92.02868938446045) proc.memory.availableMB‚ 1306.01953125 proc.memory.rssMB‚3013.625 disk.sdb1.out‚0" disk.sda2.out‚1274.18115234375 disk.nvme1n1.in‚ 744.57421875 proc.cpu.threads‚17 network.sent‚ 2.5299031e+07) disk./.usagePercent‚75.77689857027103 disk.dm-0.in‚ 1684.23828125 disk.rbd2.out‚ 3.537109375 network.recv‚ 3.716396e+06 disk.rbd0.in‚0 disk.md0p1.out‚ 1225.58984375 disk.md0.out‚ 1225.58984375 disk.rbd2.in‚0 disk.sda.in‚8.515625! disk.sda.out‚1274.18115234375 disk.sdb1.in‚0! disk.sdb.out‚1274.18115234375! disk.dm-0.out‚164843.01171875" disk.sdb2.out‚1274.18115234375 disk.sda1.in‚0 disk.md0p1.in‚ 12.296875 disk.sdb2.in‚3.78125[ü_¿4¦(/ _wandbruntime‚11415  _runtime‚11415±É§(à ' lr-AdamW/pg1‚0.00012535686463051632 ' lr-AdamW/pg2‚0.00012535686463051632 trainer/global_step‚9649 $ _timestamp‚1.788507357791649e+09 _runtime‚11415.568946929 _step‚789Ãc Ψ(È ' lr-AdamW/pg1‚0.00012535686463051632 ' lr-AdamW/pg2‚0.00012535686463051632 trainer/global_step‚9649 $ _timestamp‚1.788507357791649e+09 _runtime‚11415.568946929 _step‚789•¼¯6C4©(/ _wandbruntime‚11415  _runtime‚11415Ñ‘VC ª(P UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚822617726291968 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 244156103 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚795778898227691 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚822618654266368 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 244156103 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚822618654266368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 201379708 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚907040969272832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11527697453056 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 201379708 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚790 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚795779823242055 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10999363876123 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 179019758 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 179019139 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58367999 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218296320 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11000271851127 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11041559271936 D>DeviceSttL#9ÄatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚11415.569860104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚895513053523456 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11040649300480 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11527915749376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 201379710 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 120651142 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚907040727036416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11041559271936 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11041559271936 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚11415.569860104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚895513053523456 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11040649300480 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11527915749376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 201379710 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 120651142 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚907040727036416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11041559271936 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚822617726291968 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 244156103 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚795778898227691 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚822618654266368 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 244156103 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚822618654266368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 201379708 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚907040969272832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11527697453056 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 201379708 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚790 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944–¸Mõ4¬(/ _wandbruntime‚11416  _runtime‚11416×_O'­(‹N REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58373763 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885073589490826e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 201399868 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚811658696077312 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚784858551946112 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚822701394825728 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 445607174 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚811658714080256 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 244180934 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚631 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11000481149401 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚795859958110085 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11001389124613 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 445607174 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58374394 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 120663207 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11528850393088 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 244207304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 _runtime‚11416.728237486 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 201399868 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚784858568985472 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 179036968 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚907132366778368 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11042680745472 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚795859033095513 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 _step‚791 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 201399870 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 244207304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 _runtime‚11416.728237486 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 201399868 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚784858568985472 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 179036968 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚907132366778368 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11042680745472 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚795859033095513 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 _step‚791 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 201399870 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚811658696077312 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚784858551946112 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚822701394825728 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 445607174 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚811658714080256 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 244180934 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚631 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11000481149401 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚795859958110085 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11001389124613 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 445607174 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58374394 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 120663207 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11528850393088 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393—L_4p4¯(/ _wandbruntime‚11416  _runtime‚11416¡3cÔp°(Ö 1loss/FBt1/assignment_loss‚0.13319646848910704 0loss/FBt4/assignment_loss‚0.2791755935670608 1loss/SRqqt4/detection_loss‚2.6341051842485155 /loss/FBt3/detection_loss‚2.0162828394344876 0loss/FRt1/assignment_loss‚2.3331524701345536 1loss/SRqqt1/detection_loss‚2.2057089433073997 1loss/SRqqt3/detection_loss‚0.8236911533896228 $ _timeú|Qìstamp‚1.788507358952456e+09 0loss/FRt3/assignment_loss‚1.1878310450485774 2loss/SRqqt4/assignment_loss‚1.4501371954434685 1loss/FBt3/assignment_loss‚0.12038710771594197 %loss/total_loss‚40.78088357615626 2loss/SRqqt3/assignment_loss‚0.8390386762338684 .loss/FRt3/detection_loss‚1.474800787546805 _runtime‚11416.729694382 0loss/FRt2/assignment_loss‚3.3330556113144447 2loss/SRqqt1/assignment_loss‚2.1765769397219024 0loss/FBt2/assignment_loss‚0.2535886245952194 /loss/FRt4/detection_loss‚2.9495378486041366 0loss/SRqqt2/detection_loss‚2.342761293053627 .loss/FBt4/detection_loss‚2.033271324634552 .loss/FBt2/detection_loss‚2.382437398036321 epoch‚7 _step‚792 2loss/SRqqt2/assignment_loss‚1.8227804232543956 /loss/FRt1/detection_loss‚1.8750511375993657 /loss/FRt2/detection_loss‚2.7924859388891994 /loss/FRt4/assignment_loss‚2.532836572877292 /loss/FBt1/detection_loss‚0.7889929990163974 trainer/global_step‚9649¥þñ8a±(Û 2loss/SRqqt2/assignment_loss‚1.8227804232543956 /loss/FRt1/detection_loss‚1.8750511375993657 /loss/FRt2/detection_loss‚2.7924859388891994 /loss/FRt4/assignment_loss‚2.532836572877292 /loss/FBt1/detection_loss‚0.7889929990163974 trainer/global_step‚9649 1loss/FBt1/assignment_loss‚0.13319646848910704 0loss/FBt4/assignment_loss‚0.2791755935670608 1loss/SRqqt4/detection_loss‚2.6341051842485155 /loss/FBt3/detection_loss‚2.0162828394344876 0loss/FRt1/assignment_loss‚2.3331524701345536 1loss/SRqqt1/detection_loss‚2.2057089433073997 1loss/SRqqt3/detection_loss‚0.8236911533896228 $ _timestamp‚1.788507358952456e+09 0loss/FRt3/assignment_loss‚1.1878310450485774 2loss/SRqqt4/assignment_loss‚1.4501371954434685 1loss/FBt3/assignment_loss‚0.12038710771594197 %loss/total_loss‚40.78088357615626 2loss/SRqqt3/assignment_loss‚0.8390386762338684 .loss/FRt3/detection_loss‚1.474800787546805 _runtime‚11416.729694382 0loss/FRt2/assignment_loss‚3.3330556113144447 2loss/SRqqt1/assignment_loss‚2.1765769397219024 0loss/FBt2/assignment_loss‚0.2535886245952194 /loss/FRt4/detection_loss‚2.9495378486041366 0loss/SRqqt2/detection_loss‚2.342761293053627 .loss/FBt4/detection_loss‚2.033271324634552 .loss/FBt2/detection_loss‚2.382437398036321 epoch‚7 _step‚792˜ÓOœ²(‚(:‘ ìééÔÊÌm gpu.0.gpu‚27.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚117.81' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.452499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰Tߨj³(‚(:ß ìééÔ—™ºn network.sent‚ 2.5315668e+07 disk.sda2.in‚8.515625 disk.nvme1n1.in‚ 744.57421875& proc.memory.availableMB‚ 1306.03125 proc.cpu.threads‚17$ disk.nvme0n1.out‚116123.61328125 disk.rbd2.out‚ 3.537109375* proc.memory.percent‚18.393707275390625 disk.rbd0.in‚0 disk.md0.in‚ 12.296875 disk.md0.out‚ 1227.04296875 disk.rbd2.in‚0 disk.sda.in‚8.515625 proc.memory.rssMB‚3013.625) disk./.usagePercent‚75.78039778234836 disk.sda1.in‚0 disk.nvme0n1.in‚ 939.6640625 disk.sdb2.in‚3.78125 disk.sda1.out‚0 disk.md0p1.in‚ 12.296875 disk.sdb.out‚1275.6845703125$ memory_percent‚92.02861785888672 cpu‚25.187769525925603 network.recv‚ 3.720946e+06$ disk./.usageGB‚2710.274085998535 disk.sdb1.in‚0 disk.md0p1.out‚ 1227.04296875 disk.dm-0.in‚ 1684.23828125 disk.sdb.in‚3.78125! disk.nvme1n1.out‚ 48900.640625 disk.sda.out‚1275.6845703125! disk.dm-0.out‚165024.25390625! disk.sda2.out‚1275.6845703125 disk.rbd0.out‚ 3.025390625 disk.sdb1.out‚0! disk.sdb2.out‚1275.6845703125£XC“´(‚(:ˆ ûééÔýä˜m gpu.0.gpu‚65.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚186.707' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚46.67675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§Œ Énµ(‚(:ã ûééÔÅÃn% disk./.usageGB‚2710.8986740112305# disk.nvme1n1.out‚49124.84765625 disk.md0.out‚ 1228.93359375 disk.rbd2.out‚ 3.537109375 disk.sda1.in‚0 disk.nvme0n1.in‚ 939.6640625 disk.md0.in‚ 12.296875" disk.sdb2.out‚1277.62939453125 disk.sdb1.in‚0 disk.md0p1.in‚ 12.296875 proc.memory.rssMB‚3013.625! disk.sda.out‚1277.62939453125 disk.md0p1.out‚ 1228.93359375 disk.sdb.in‚3.78125 cpu‚25.18785290513268 proc.cpu.threads‚17( disk./.usagePercent‚75.7978615245937 disk.rbd2.in‚0 disk.rbd0.out‚ 3.025390625 disk.sdb1.out‚0 disk.dm-0.in‚ 1684.23828125 disk.sda2.in‚8.515625" disk.sda2.out‚1277.62939453125! disk.sdb.out‚1277.62939453125 disk.nvme1n1.in‚ 744.57421875 disk.sda1.out‚0 disk.rbd0.in‚0 network.sent‚ 2.5412564e+07 disk.sda.in‚8.515625 disk.dm-0.out‚165248.4609375$ memory_percent‚92.02861785888672$ disk.nvme0n1.out‚116123.61328125 disk.sdb2.in‚3.78125 network.recv‚ 3.726889e+06& proc.memory.availableMB‚ 1306.03125* proc.memory.percent‚18.393707275390625EQ럒¶(‚(:‡ ŠêéÔ™ØÉk gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.181' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.29525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y!¸oo·(‚(:ä ŠêéÔ£ô‘m) proc.memory.availableMB‚ 1306.04296875" disk.sdb2.out‚1279.08154296875 disk.sda1.in‚0 disk.dm-0.in‚ 1684.23828125 disk.dm-0.out‚ 165345.6875 disk.md0.out‚ 1230.32421875 cpu‚25.192378675875617 network.sent‚ 2.5416639e+07! disk.sda.out‚1279.08154296875 disk.rbd0.in‚0" disk.sda2.out‚1279.08154296875 disk.nvme1n1.in‚ 744.57421875 disk.rbd0.out‚ 3.025390625 disk.md0p1.in‚ 12.296875$ memory_percent‚92.02854633331299% disk./.usageGB‚2711.1486740112305 disk.md0p1.out‚ 1230.32421875! disk.sdb.out‚1279.08154296875 disk.sda2.in‚8.515625# disk.nvme1n1.out‚49219.99609375 disk.md0.in‚ 12.296875 disk.rbd2.out‚ 3.537109375 proc.cpu.threads‚17 disk.sdb2.in‚3.78125 disk.sda.in‚8.515625$ disk.nvme0n1.out‚116125.69140625* proc.memory.percent‚18.393707275390625) disk./.usagePercent‚75.80485162922683 disk.sdb1.out‚0 disk.sdb.in‚3.78125 disk.rbd2.in‚0 network.recv‚ 3.73177e+06 disk.sda1.out‚0 disk.sdb1.in‚0 disk.nvme0n1.in‚ 939.6640625 proc.memory.rssMB‚3013.625¹ù¼‘¸(‚(:† ™êéÔè°Þk gpu.0.gpu‚9.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.118' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.0295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%@jûi¹(‚(:Þ ™êéÔ¹ôÛm disk.rbd2.out‚ 3.537109375 disk.sdb1.out‚0 disk.md0p1.in‚ 12.296875 network.sent‚ 2.5424108e+07 disk.dm-0.out‚165691 disk.nvme0n1.in‚ 939.6640625 disk.md0p1.out‚ 1231.77734375 disk.dm-0.in‚ 1684.23828125 network.recv‚ 3.73517e+06) disk./.usagePercent‚75.80486474847228 disk.sda1.in‚0" disk.sda2.out‚1280.59814453125$ disk.nvme0n1.out‚116125.69140625 proc.cpu.threads‚17 disk.md0.out‚ 1231.77734375$ memory_percent‚92.03016757965088) proc.memory.availableMB‚ 1305.77734375" disk.sdb2.out‚1280.59814453125 disk.sda.in‚8.515625! disk.sda.out‚1280.59814453125 proc.memory.rssMB‚3013.625 disk.sdb2.in‚3.78125 disk.sda1.out‚0 disk.sdb.in‚3.78125 disk.md0.in‚ 12.296875 disk.rbd0.out‚ 3.025390625 disk.sdb1.in‚0! disk.sdb.out‚1280.59814453125# disk.nvme1n1.out‚49565.30859375* proc.memory.percent‚18.393707275390625 cpu‚25.197342333470136 disk.sda2.in‚8.515625 disk.nvme1n1.in‚ 744.57421875 disk.rbd0.in‚0$ disk./.usageGB‚2711.149143218994 disk.rbd2.in‚0Òæ³=“º(‚(:ˆ ¨êéÔóúl gpu.0.gpu‚51.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚127.779' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.94475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BÊxqp»(‚(:å ¨êéÔì˲n disk.rbd2.in‚0! disk.sda.out‚1281.97900390625 disk.dm-0.in‚ 1684.23828125 disk.sda2.in‚8.515625 disk.md0.in‚ 12.296875 disk.sdb2.in‚3.78125* proc.memory.percent‚18.393707275390625 disk.rbd2.out‚ 3.537109375 disk.md0p1.in‚ 12.296875" disk.sdb2.out‚1281.97900390625 disk.sda1.out‚0 disk.md0p1.out‚ 1233.1015625" disk.sda2.out‚1281.97900390625 disk.nvme0n1.in‚ 939.6640625 disk.md0.out‚ 1233.1015625! disk.sdb.out‚1281.97900390625$ disk.nvme0n1.out‚116125.91796875$ memory_percent‚92.03107357025146 disk.sda.in‚8.515625 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 3.025390625 disk.sdb1.in‚0! disk.dm-0.out‚165691.64453125" disk.nvme1n1.out‚ 49565.7265625 proc.memory.rssMB‚3013.625 network.sent‚ 2.5430251e+07) disk./.usagePercent‚75.80276374931054 disk.sdb.in‚3.78125 disk.nvme1n1.in‚ 744.57421875) proc.memory.availableMB‚ 1305.62890625 cpu‚25.202466363318674 proc.cpu.threads‚17$ disk./.usageGB‚2711.074001312256 disk.sdb1.out‚0 network.recv‚ 3.739474e+06pùº~4¼(/ _wandbruntime‚11504  _runtime‚11504DÀÃn*½(èT ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚800117746233369 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚11091373905408 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚58766879 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 447787298 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚11091373905408 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 202601712 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 245164873 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 180164052 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚911924600706048 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚900345330921472 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14401142784 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 14135738368 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚816113118411776 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚789067818180096 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14401142784 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189768192 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 121397175 % _timestamp‚1.7885074471243873e+09 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 447766585 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 174695936 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1185 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚570 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 447787298 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 245185584 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 180164280 ZLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚ 121397173 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚615 _runtime‚11504.902532953 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1185 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14401142784 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚816113100408832 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚9671 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚11090690866176 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 198636032 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚789067835219456 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚827204492317184 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚827204492317184 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 701042176 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1231 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚900345354861568 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚615 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚827203791275008 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚816113118411776 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13684416521 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 202601714 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 698106808 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1185 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚570 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 202601712 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3725 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249a©‚ìs*¾(íT jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚900345354861568 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚615 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚827203791275008 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚816113118411776 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13684416521 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 202601714 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 698106808 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1185 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚570 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 202601712 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3725 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚800117746233369 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚11091373905408 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚58766879 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 447787298 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚11091373905408 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 202601712 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 245164873 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 180164052 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚911924600706048 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚900345330921472 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14401142784 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 14135738368 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚816113118411776 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚789067818180096 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14401142784 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189768192 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 121397175 % _timestamp‚1.7885074471243873e+09 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 447766585 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 174695936 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1185 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚570 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 447787298 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 245185584 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 180164280 ZLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚ 121397173 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚615 _runtime‚11504.902532953 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1185 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14401142784 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚816113100408832 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚9671 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚11090690866176 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 198636032 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚789067835219456 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚827204492317184 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚827204492317184 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 701042176 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1231 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615™Û%8O’¿(‚(:‡ ·êéÔ’Èm gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.181' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.29525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à·À(‚(:Ü ·êéÔ¦ÞÈn$ disk.nvme0n1.out‚116125.91796875# disk.nvme1n1.out‚49901.12890625 cpu‚25.207622567324346 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 3.025390625 disk.md0p1.in‚ 12.296875 proc.memory.rssMB‚3013.625 proc.cpu.threads‚17 disk.dm-0.in‚ 1684.23828125 disk.sdb.in‚3.78125 disk.nvme1n1.in‚ 744.57421875$ memory_percent‚92.˜=Wq×03174114227295( disk./.usagePercent‚75.7901127435956 disk.sda1.in‚0 disk.sda2.out‚1283.923828125) proc.memory.availableMB‚ 1305.51953125 network.sent‚ 2.5446695e+07 disk.rbd2.in‚0 disk.md0p1.out‚ 1234.98828125 disk.dm-0.out‚ 166027.046875* proc.memory.percent‚18.393707275390625 network.recv‚ 3.743537e+06 disk.rbd2.out‚ 3.537109375 disk.sdb.out‚1283.923828125 disk.sda2.in‚8.515625# disk./.usageGB‚2710.62154006958 disk.nvme0n1.in‚ 939.6640625 disk.md0.out‚ 1234.98828125 disk.sdb2.out‚1283.923828125 disk.sda.in‚8.515625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.in‚ 12.296875 disk.sdb2.in‚3.78125 disk.sda.out‚1283.923828125@.Ù®4Á(/ _wandbruntime‚11506  _runtime‚11506>¢vx)Â(òR XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚800167962442629 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚789118261324640 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 _runtime‚11506.121010361 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 447790875 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1268 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚911981358110720 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 681067448 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚800168660549437 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3725 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 245173687 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1185 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 245173687 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135738368 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚900401622415360 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚827258542032896 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 447811588 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚816167379318784 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 683039232 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚789118244285280 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 409073152 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚11049718157349 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚816167361315840 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚816167379318784 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 198636032 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7885074483428242e+09 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚615 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚227 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1195376640 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚227 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1195376640 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 B&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 epoch‚7 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4712391793727875 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074074625968933 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.48489606380462646 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5747939944267273 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9946901202201843 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.20881497859954834 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5027451515197754 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.16580457985401154 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,jet/accuracy_3_of_3‚0.027103982865810394 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1388888955116272 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 _step‚795 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0873015895485878 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010275604203343391 jet/accuracy_7_of_7‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21018098294734955 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3057263195514679 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.47141340374946594 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14840547740459442 particle/accuracy_3_of_3‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1358497142791748 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4270833432674408 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.36089327931404114 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 8"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.803030252456665 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09156379103660583 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2331974357366562 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.6000000238418579 jet/accuracy_6_of_7‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993895292282104 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2232142835855484 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.005286886356770992 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.17424243688583374 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24425287544727325 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9769211411476135 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 *jet/accuracy_1_of_2‚0.7501093745231628 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10352656990289688 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.6020753979682922 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4308428466320038 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 6validation_average_jet_accuracy‚0.4313037097454071 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3850763738155365 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.5 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_5_of_5‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6455432176589966 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2841467261314392 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0416666679084301 *jet/accuracy_2_of_6‚0.6025640964508057 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2858092784881592 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.053468793630599976 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_6‚0.9615384340286255 jet/accuracy_4_of_7‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17971310019493103 +jet/accuracy_2_of_2‚0.11071375757455826 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.0602012537419796 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3889402151107788 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14985033869743347 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_6_of_6‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9978200793266296 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4874779284000397 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09914013743400574 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9864341616630554 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3886904716491699 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.24166664481163025 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqjøu,Úoqt_purity‚0 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.997442901134491 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1403484046459198 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1621924787759781 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.32902511954307556 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 *jet/accuracy_1_of_5‚0.9056515693664551 particle/accuracy_1_of_4‚1 particle/accuracy_5_of_6‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9399999976158142 jet/accuracy_3_of_7‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2455948442220688 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.00833333283662796 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.49574166536331177 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.482963889837265 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5739011168479919 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_2_of_2‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08865249156951904 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5137526988983154 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6045881509780884 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2648148238658905 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1660628765821457 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.34535306692123413 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1388888955116272 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5570799112319946 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.48886916041374207 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 8"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.418226033449173 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5790202617645264 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4851114749908447 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.028106460347771645 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5998899340629578 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9905070066452026 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.02358490601181984 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4699612557888031 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.14678363502025604 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.5185185074806213 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1388888955116272 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4220302999019623 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 particle/accuracy_5_of_7‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4604748487472534 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.15645946562290192 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_6‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5168623924255371 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.024622609838843346 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16621296107769012 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998443067073822 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15847738087177277 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3860393762588501 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5171850323677063 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5016629695892334 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.38977617025375366 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23430533707141876 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.49662667512893677 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.46225038170814514 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14390771090984344 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9748427867889404 % _timestamp‚1.7885074643095171e+09 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.45485857129096985 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.37827086448669434 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.17870211601257324 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9770114421844482 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23607179522514343 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5416666865348816 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.41235122084617615 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6696428656578064 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5432099103927612 particle/accuracy_6_of_7‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17615428566932678 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.996999979019165 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8076120018959045 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20018351078033447 :"Purity/*FRt3SRqqt1FBt/event_purity‚0.03703703731298447 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6578947901725769 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5952090620994568 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15052004158496857 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5416666865348816 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_7‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11389343440532684 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.48601746559143066 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.6115902662277222 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.46481752395629883 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5545977354049683 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_4_of_7‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5061640739440918 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2015303373336792 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0706845223903656 particle/accuracy_1_of_3‚1 +jet/accuracy_3_of_6‚0.12820512056350708 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.48315921425819397 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23059597611427307 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18370923399925232 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9908846616744995 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5127893090248108 +jet/accuracy_3_of_5‚0.11721069365739822 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02239503338932991 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5427631735801697 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4850933849811554 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9916394948959351 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5354517698287964 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1286313533782959 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5356031656265259 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1802261769771576 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3750842213630676 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5416666865348816 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 'particle/f_score‚0.4204459488391876 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9876500368118286 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 particle/accuracy_3_of_4‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5393518805503845 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17550504207611084 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9821428656578064 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9911333322525024 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6111111044883728 particle/accuracy_1_of_5‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3333333432674408 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.026657162234187126 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5566666722297668 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5400089621543884 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9629628658294678 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt3SRqqt1FBt/event_purity‚0.1111111119389534 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9887440204620361 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5758744478225708 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ;"Purity/0FRt*SRqqt3FBt/event_purity‚0.048245616257190704 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.47141340374946594 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.07015830278396606 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.15111109614372253 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9903015494346619 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.24664124846458435 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21125799417495728 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.46338680386543274 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13820582628250122 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2586762309074402 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 *jet/accuracy_1_of_4‚0.9201350212097168 ,jet/accuracy_4_of_5‚0.008602150715887547 +particle/sensitivity‚0.3388693928718567 jet/accuracy_5_of_7‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17929738759994507 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9078947305679321 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 particle/accuracy_2_of_7‚1 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2413116842508316 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9722222685813904 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 (particle/accuracy‚0.8542236685752869 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17720133066177368 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9652777314186096 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.40039679408073425 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.968254029750824 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3171500265598297 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5648704171180725 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4849787950515747 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3903219401836395 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9955723285675049 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5383638739585876 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6380800008773804 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4910815358161926 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3893393576145172 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322916865348816 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4933333098888397 jet/accuracy_1_of_7‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5018038749694824 particle/accuracy_3_of_6‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5370370149612427 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9915459156036377 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.8695903420448303 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07947809249162674 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2986111044883728 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17720133066177368 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 particle/accuracy_3_of_5‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14375294744968414 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9721102714538574 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.001792114693671465 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.382535457611084 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 ( Purity/*FRt3SRqqt2FBt/FBt_purity‚0.5 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.545258641242981 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5289492607116699 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_2‚1 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.05056434124708176 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1689189225435257 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 _runtime‚11522.088991982 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.21682099997997284 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.985417902469635 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9815316200256348 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_6‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7059857249259949 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9739229083061218 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9495878219604492 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.42222216725349426 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006191463675349951 particle/accuracy_1_of_7‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9928429126739502 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6047619581222534 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24762924015522003 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5475693941116333 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.38257578015327454 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18896694481372833 ( Purity/1FRt3SRqqt2FBt/FBt_purity‚0.5 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.991113543510437 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6052631735801697 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06885693222284317 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6298307180404663 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_6‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9255319237709045 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.019885359331965446 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5138636827468872 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3539215624332428 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3987264633178711 particle/accuracy_2_of_3‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17589783668518066 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.020365990698337555 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16233061254024506 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.0283827967941761 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2856263816356659 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012010030448436737 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.46081024408340454 6 Purity/1FRt3SRqqt*FBt/FBt_purity‚0.970588207244873 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23529396951198578 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9902504086494446 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.37707510590553284 jet/accuracy_5_of_6‚0 jet/accuracy_2_of_7‚0.5 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4764992892742157 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 trainer/global_step‚9671‰ßY<È(ö¦ :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17589783668518066 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.020365990698337555 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16233061254024506 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.0283827967941761 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2856263816356659 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012010030448436737 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.46081024408340454 6 Purity/1FRt3SRqqt*FBt/FBt_purity‚0.970588207244873 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23529396951198578 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9902504086494446 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.37707510590553284 jet/accuracy_5_of_6‚0 jet/accuracy_2_of_7‚0.5 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4764992892742157 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 trainer/global_step‚9671 )loss/val_total_loss‚40.34925740842728 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5500895977020264 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9913061261177063 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 epoch‚7 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4712391793727875 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074074625968933 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.48489606380462646 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5747939944267273 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9946901202201843 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.20881497859954834 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5027451515197754 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.16580457985401154 @&Purity/1FRt3SRqq>½¨ùt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,jet/accuracy_3_of_3‚0.027103982865810394 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1388888955116272 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 _step‚795 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0873015895485878 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010275604203343391 jet/accuracy_7_of_7‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21018098294734955 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3057263195514679 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.47141340374946594 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14840547740459442 particle/accuracy_3_of_3‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1358497142791748 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4270833432674408 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.36089327931404114 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 8"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.803030252456665 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09156379103660583 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2331974357366562 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.6000000238418579 jet/accuracy_6_of_7‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993895292282104 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2232142835855484 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.005286886356770992 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.17424243688583374 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24425287544727325 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9769211411476135 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 *jet/accuracy_1_of_2‚0.7501093745231628 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10352656990289688 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.6020753979682922 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4308428466320038 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 6validation_average_jet_accuracy‚0.4313037097454071 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3850763738155365 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.5 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_5_of_5‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6455432176589966 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2841467261314392 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0416666679084301 *jet/accuracy_2_of_6‚0.6025640964508057 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2858092784881592 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.053468793630599976 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_6‚0.9615384340286255 jet/accuracy_4_of_7‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17971310019493103 +jet/accuracy_2_of_2‚0.11071375757455826 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.0602012537419796 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3889402151107788 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14985033869743347 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_6_of_6‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9978200793266296 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4874779284000397 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09914013743400574 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9864341616630554 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3886904716491699 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.24166664481163025 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.997442901134491 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1403484046459198 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1621924787759781 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.32902511954307556 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 *jet/accuracy_1_of_5‚0.9056515693664551 particle/accuracy_1_of_4‚1 particle/accuracy_5_of_6‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9399999976158142 jet/accuracy_3_of_7‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2455948442220688 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.00833333283662796 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.49574166536331177 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.482963889837265 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5739011168479919 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_2_of_2‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08865249156951904 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5137526988983154 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6045881509780884 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2648148238658905 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1660628765821457 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.34535306692123413 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1388888955116272 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5570799112319946 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.48886916041374207 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 8"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.418226033449173 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5790202617645264 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4851114749908447 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.028106460347771645 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5998899340629578 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9905070066452026 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.02358490601181984 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4699612557888031 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.14678363502025604 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.5185185074806213 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1388888955116272 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4220302999019623 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 particle/accuracy_5_of_7‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4604748487472534 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.15645946562290192 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_6‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5168623924255371 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.024622609838843346 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16621296107769012 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998443067073822 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15847738087177277 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3860393762588501 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5171850323677063 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5016629695892334 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.38977617025375366 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23430533707141876 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.49662667512893677 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.46225038170814514 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14390771090984344 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9748427867889404 % _timestamp‚1.7885074643095171e+09 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.45485857129096985 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.37827086448669434 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.17870211601257324 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9770114421844482 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23607179522514343 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5416666865348816 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.41235122084617615 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6696428656578064 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5432099103927612 particle/accuracy_6_of_7‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17615428566932678 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.996999979019165 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8076120018959045 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20018351078033447 :"Purity/*FRt3SRqqt1FBt/event_purity‚0.03703703731298447 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6578947901725769 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5952090620994568 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15052004158496857 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5416666865348816 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_7‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11389343440532684 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.48601746559143066 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.6115902662277222 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.46481752395629883 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5545977354049683 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_4_of_7‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5061640739440918 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2015303373336792 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0706845223903656 particle/accuracy_1_of_3‚1 +jet/accuracy_3_of_6‚0.12820512056350708 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.48315921425819397 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23059597611427307 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18370923399925232 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9908846616744995 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5127893090248108 +jet/accuracy_3_of_5‚0.11721069365739822 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02239503338932991 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5427631735801697 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4850933849811554 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9916394948959351 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5354517698287964 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1286313533782959 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5356031656265259 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1802261769771576 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3750842213630676 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5416666865348816 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 'particle/f_score‚0.4204459488391876 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9876500368118286 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 particle/accuracy_3_of_4‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5393518805503845 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17550504207611084 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9821428656578064 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9911333322525024 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6111111044883728 particle/accuracy_1_of_5‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3333333432674408 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.026657162234187126 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5566666722297668 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5400089621543884 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9629628658294678 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt3SRqqt1FBt/event_purity‚0.1111111119389534 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9887440204620361 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5758744478225708 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ;"Purity/0FRt*SRqqt3FBt/event_purity‚0.048245616257190704 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.47141340374946594 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.07015830278396606 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.15111109614372253 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9903015494346619 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.24664124846458435 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21125799417495728 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.46338680386543274 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13820582628250122 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2586762309074402 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 *jet/accuracy_1_of_4‚0.9201350212097168 ,jet/accuracy_4_of_5‚0.008602150715887547 +particle/sensitivity‚0.3388693928718567 jet/accuracy_5_of_7‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17929738759994507 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9078947305679321 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 particle/accuracy_2_of_7‚1 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2413116842508316 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9722222685813904 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 (particle/accuracy‚0.8542236685752869 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17720133066177368 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9652777314186096 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.40039679408073425 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.968254029750824 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3171500265598297 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5648704171180725 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4849787950515747 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3903219401836395 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9955723285675049 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5383638739585876 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6380800008773804 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4910815358161926 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3893393576145172 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322916865348816 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4933333098888397 jet/accuracy_1_of_7‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5018038749694824 particle/accuracy_3_of_6‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5370370149612427 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9915459156036377 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.8695903420448303 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07947809249162674 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2986111044883728 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17720133066177368 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 particle/accuracy_3_of_5‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14375294744968414 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9721102714538574 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.001792114693671465 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.382535457611084 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 ( Purity/*FRt3SRqqt2FBt/FBt_purity‚0.5 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.545258641242981 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5289492607116699 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_2‚1 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.05056434124708176 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1689189225435257 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 _runtime‚11522.088991982 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.21682099997997284 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.985417902469635 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9815316200256348 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_6‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7059857249259949 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9739229083061218 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9495878219604492 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.42222216725349426 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006191463675349951 particle/accuracy_1_of_7‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9928429126739502 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6047619581222534 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24762924015522003 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5475693941116333 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.38257578015327454 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18896694481372833 ( Purity/1FRt3SRqqt2FBt/FBt_purity‚0.5 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.991113543510437 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6052631735801697 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06885693222284317 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purP†ÝÒlity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6298307180404663 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_6‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9255319237709045 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.019885359331965446 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5138636827468872 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3539215624332428 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3987264633178711 particle/accuracy_2_of_3‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0›õ Š†“É(‚(:ˆ ÕêéÔÿñôm gpu.0.gpu‚35.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.637' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.15925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆºÑnÊ(‚(:ã ÕêéÔŽ‡Ún cpu‚25.213505943755784% disk./.usageGB‚2710.6492195129395 disk.rbd2.in‚0 disk.md0.out‚ 1237.88671875 proc.memory.rssMB‚3013.625" disk.nvme1n1.out‚ 50016.8984375 disk.sda1.out‚0 disk.rbd0.in‚0! disk.sda2.out‚1286.9521484375$ disk.nvme0n1.out‚116203.01171875 proc.cpu.threads‚17) disk./.usagePercent‚75.79088667241666 disk.sda.in‚8.515625 disk.sdb1.in‚0 disk.dm-0.in‚ 1684.23828125 disk.sda2.in‚8.515625 disk.nvme0n1.in‚ 939.6640625 disk.rbd0.out‚ 3.025390625 disk.sdb1.out‚0 disk.sdb.out‚1286.9521484375* proc.memory.percent‚18.393707275390625 network.recv‚ 3.754447e+06! disk.sdb2.out‚1286.9521484375 disk.md0p1.in‚ 12.296875( proc.memory.availableMB‚ 1305.6796875 network.sent‚ 2.5639489e+07 disk.rbd2.out‚ 3.537109375 disk.sda.out‚1286.9521484375! disk.dm-0.out‚166219.91015625$ memory_percent‚92.03076362609863 disk.sdb2.in‚3.78125 disk.sdb.in‚3.78125 disk.nvme1n1.in‚ 744.57421875 disk.sda1.in‚0 disk.md0p1.out‚ 1237.88671875 disk.md0.in‚ 12.296875N +lË(‚(:á äêéÔÚ¯m" disk.sdb2.out‚1288.57080078125$ memory_percent‚92.03076362609863 network.recv‚ 3.760656e+06 disk.dm-0.out‚166481.5078125 disk.nvme0n1.in‚ 939.6640625$ disk.nvme0n1.out‚116292.26171875 disk.rbd0.out‚ 3.025390625( proc.memory.availableMB‚ 1305.6796875* proc.memory.percent‚18.393707275390625 disk.sdb.in‚3.78125! disk.sdb.out‚1288.57080078125 disk.sda.in‚8.515625 disk.rbd0.in‚0 disk.md0p1.in‚ 12.296875" disk.sda2.out‚1288.57080078125 disk.md0.in‚ 12.296875 disk.sda1.in‚0 proc.memory.rssMB‚3013.625 network.sent‚ 2.5752436e+07% disk./.usageGB‚2710.8990058898926 disk.rbd2.out‚ 3.537109375! disk.sda.out‚1288.57080078125 disk.sda1.out‚0 cpu‚25.21325695324096 proc.cpu.threads‚17 disk.dm-0.in‚ 1684.23828125 disk.nvme1n1.in‚ 744.57421875 disk.sdb1.in‚0 disk.md0p1.out‚ 1239.4453125 disk.rbd2.in‚0 disk.sdb1.out‚0 disk.sda2.in‚8.515625# disk.nvme1n1.out‚50189.24609375 disk.md0.out‚ 1239.4453125& disk./.usagePercent‚75.79787080406 disk.sdb2.in‚3.78125æÎy|‘Ì(‚(:† äêéÔ«¢Öm gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.798' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.4495 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡ ¶ë4Í(/ _wandbruntime‚11554  _runtime‚11554rfã‹ÊÎ(Ä ' lr-AdamW/pg1‚0.00012047767723303172 ' lr-AdamW/pg2‚0.00012047767723303172 trainer/global_step‚9699 % _timestamp‚1.7885074969733775e+09 _runtime‚11554.750677309 _step‚796;~_HÏÏ(É % _timestamp‚1.7885074969733775e+09 _runtime‚11554.750677309 _step‚796 ' lr-AdamW/pg1‚0.00012047767723303172 ' lr-AdamW/pg2‚0.00012047767723303172 trainer/global_step‚9699œü¿'M4Ð(/ _wandbruntime‚11554  _runtime‚11554‘o©q(Ñ(€P QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 180851716 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 245958219 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 203362696 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚791907258705360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚915186918900736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚903569170694656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235073536 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚802994199244020 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 203362696 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚830204082743296 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11617724265984 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9699 $ _timestamp‚1.788507496974093e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 449320915 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 245984563 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 449347261 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚819076486427136 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11128524290560 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚830204082743296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58994290 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚791907275744720 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚819076468424192 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11127614319104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚628 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚903569194634752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259013632 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 203362698 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11127614319104 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 245984563 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚797 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚830205010717696 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚830205010717696 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570C(D— (Ò(…P JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚830205010717696 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 180851716 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 245958219 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 203362696 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚791907258705360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚915186918900736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚903569170694656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235073536 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚802994199244020 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 203362696 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚830204082743296 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11617724265984 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9699 $ _timestamp‚1.788507496974093e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 449320915 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 245984563 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 449347261 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚819076486427136 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11128524290560 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚830204082743296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58994290 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚791907275744720 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚819076468424192 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11127614319104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚628 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚903569194634752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259013632 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 203362698 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11127614319104 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 245984563 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚797 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚830205010717696È 4Ó(/ _wandbruntime‚11555  _runtime‚11555 ñ;3õÔ(ŠN H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 449392278 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59000566 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 121868861 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 203382856 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11087132797574 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 449392278 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚819158105512960 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 449365906 _runtime‚11555.821301118 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11129645764096 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 180868809 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 _step‚798 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 245983050 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚903659396399104 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11128735779328 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 203382858 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58999948 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 $ _timestamp‚1.788507498042251e+09 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚803073409097478 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚791986293339264 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚830287751277056 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚830287751277056 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 245983050 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11618663621120 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615z²_¶'Õ(N REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58999948 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 $ _timestamp‚1.788507498042251e+09 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚803073409097478 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚791986293339264 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚830287751277056 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚830287751277056 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 245983050 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11618663621120 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 449392278 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59000566 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 121868861 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 203382856 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11087132797574 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 449392278 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚819158105512960 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 449365906 _runtime‚11555.821301118 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11129645764096 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 180868809 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 _step‚798 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 245983050 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚903659396399104 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11128735779328 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 203382858ž]/ït4Ö(/ _wandbruntime‚11555  _runtime‚11555éôÝÜ[×(Õ 0loss/FRt1/assignment_loss‚1.9322703785366482 1loss/SRqqt1/detection_loss‚1.8122327540602003 % _timestamp‚1.7885074980457034e+09 0loss/FRt3/assignment_loss‚1.1754842205481095 2loss/SRqqt4/assignment_loss‚1.9261598410084844 1loss/FBt3/assignment_loss‚0.08485718544596405 /loss/SRqqt4/detection_loss‚2.21203276515007 0loss/FRt2/assignment_loss‚3.2494760707572654 1loss/FBt1/assignment_loss‚0.15188723113893654 /loss/FRt2/detection_loss‚2.7751420083697194 .loss/FRt3/detection_loss‚1.915481094067747 2loss/SRqqt2/assignment_loss‚1.1711383618252433 .loss/FRt1/detection_loss‚2.072430220971819 1loss/SRqqt3/detection_loss‚0.7602101555164147 2loss/SRqqt1/assignment_loss‚1.2750399009989841 /loss/FRt4/detection_loss‚2.7821195942384227 .loss/FBt4/detection_loss‚2.238947865035799 _step‚799 1loss/SRqqt3/assignment_loss‚0.929263559440536 0loss/FBt4/assignment_loss‚0.4120379070111603 0loss/SRqqt2/detection_loss‚2.404561242331629 /loss/FBt1/detection_loss‚0.5815206618110339 0loss/FRt4/assignment_loss‚2.2742154664463468 1loss/FBt2/assignment_loss‚0.32611014719144515 .loss/FBt3/detection_loss‚2.189949382435192 epoch‚8 trainer/global_step‚9699 /loss/FBt2/detection_loss‚1.8790523629439504 %loss/total_loss‚38.53162037728112 _runtime‚11555.822838805œ~¥`Ø(Ú /loss/FBt1/detection_loss‚0.5815206618110339 0loss/FRt4/assignment_loss‚2.2742154664463468 1loss/FBt2/assignment_loss‚0.32611014719144515 .loss/FBt3/detection_loss‚2.189949382435192 epoch‚8 trainer/global_step‚9699 /loss/FBt2/detection_loss‚1.8790523629439504 %loss/total_loss‚38.53162037728112 _runtime‚11555.822838805 0loss/FRt1/assignment_loss‚1.9322703785366482 1loss/SRqqt1/detection_loss‚1.8122327540602003 % _timestamp‚1.7885074980457034e+09 0loss/FRt3/assignment_loss‚1.1754842205481095 2loss/SRqqt4/assignment_loss‚1.9261598410084844 1loss/FBt3/assignment_loss‚0.08485718544596405 /loss/SRqqt4/detection_loss‚2.21203276515007 0loss/FRt2/assignment_loss‚3.2494760707572654 1loss/FBt1/assignment_loss‚0.15188723113893654 /loss/FRt2/detection_loss‚2.7751420083697194 .loss/FRt3/detection_loss‚1.915481094067747 2loss/SRqqt2/assignment_loss‚1.1711383618252433 .loss/FRt1/detection_loss‚2.072430220971819 1loss/SRqqt3/detection_loss‚0.7602101555164147 2loss/SRqqt1/assignment_loss‚1.2750399009989841 /loss/FRt4/detection_loss‚2.7821195942384227 .loss/FBt4/detection_loss‚2.238947865035799 _step‚799 1loss/SRqqt3/assignment_loss‚0.929263559440536 0loss/FBt4/assignment_loss‚0.4120379070111603 0loss/SRqqt2/detection_loss‚2.404561242331629ŸtXˆÍ“Ù(‚(:ˆ óêéÔšÁn gpu.0.gpu‚29.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚154.735' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.68375 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿ê±uTÚ(‚(:É óêéÔḌo network.sent‚ 2.576888e+07 disk.rbd2.out‚ 3.537109375 disk.rbd0.out‚ 3.025390625' proc.memory.availableMB‚ 1304.671875 cpu‚25.21299496394688 disk.nvme1n1.in‚ 744.57421875 disk.sda.in‚8.515625 disk.sda2.out‚ 1290.4765625$ memory_percent‚92.03691482543945 network.recv‚ 3.764774e+06# disk.nvme1n1.out‚50189.43359375 disk.sdb1.out‚0 disk.md0p1.in‚ 12.296875 disk.dm-0.in‚ 1684.23828125 disk.dm-0.out‚ 166541.75 proc.memory.rssMB‚3013.625* proc.memory.percent‚18.393707275390625 disk.nvme0n1.in‚ 939.6640625 disk.sdb2.out‚ 1290.4765625 disk.sda.out‚ 1290.4765625 disk.sdb1.in‚0 disk.sdb.out‚ 1290.4765625 disk.sda2.in‚8.515625) disk./.usagePercent‚75.80492874479155 disk.md0.out‚ 1241.28125 disk.sda1.in‚0 disk.md0p1.out‚ 1241.28125 disk.sdb.in‚3.78125$ disk.nvme0n1.out‚116352.31640625 disk.md0.in‚ 12.296875 disk.rbd2.in‚0 disk.sdb2.in‚3.78125 disk.sda1.out‚0% disk./.usageGB‚2711.1514320373535 disk.rbd0.in‚0 proc.cpu.threads‚17݇T»kÛ(‚(:à ‚ëéÔ¾ùl$ disk./.usageGB‚2710.092716217041 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.md0p1.in‚ 12.296875 disk.dm-0.in‚ 1684.23828125 disk.sdb.in‚3.78125 disk.sda2.in‚8.515625 disk.nvme1n1.in‚ 744.57421875! disk.sda.out‚1291.93115234375 disk.md0.in‚ 12.296875 disk.rbd2.in‚0$ memory_percent‚92.03581809997559 disk.md0p1.out‚ 1242.671875! disk.sdb.out‚1291.93115234375" disk.sda2.out‚1291.93115234375$ disk.nvme0n1.out‚116494.75390625 disk.rbd2.out‚ 3.537109375) disk./.usagePercent‚75.77532660734863 disk.sda1.out‚0! disk.nvme1n1.out‚ 50309.859375 disk.md0.out‚ 1242.671875" disk.sdb2.out‚1291.93115234375 proc.cpu.threads‚17 disk.sdb1.out‚0! disk.dm-0.out‚166804.61328125( proc.memory.availableMB‚ 1304.8515625 proc.memory.rssMB‚3013.625 disk.sda.in‚8.515625 disk.rbd0.out‚ 3.025390625 cpu‚25.212750662894653 network.sent‚ 2.586414e+07 disk.sdb2.in‚3.78125 network.recv‚ 3.771143e+06 disk.sda1.in‚0 disk.nvme0n1.in‚ 939.6640625* proc.memory.percent‚18.393707275390625'›‘Ü(‚(:† ‚ëéÔœ«Œm gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚159.696' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.924 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qc5fÝ(‚(:Û ‘ëéÔ‘Ææl disk.sdb1.in‚0' proc.memory.availableMB‚ 1304.734375* proc.memory.percent‚18.393707275390625 disk.sda1.out‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 1684.23828125 disk.nvme0n1.in‚ 939.6640625 cpu‚25.212519755819116 disk.sda1.in‚0$ disk.nvme0n1.out‚116696.77734375 network.recv‚ 3.774787e+06 disk.sda.in‚8.515625 disk.rbd0.in‚0 disk.md0p1.in‚ 12.296875 disk.sdb.in‚3.78125! disk.sda2.out‚1293.4775390625 proc.memory.rssMB‚3013.625( disk./.usagePercent‚75.7753135947637 disk.rbd2.out‚ 3.537109375 disk.sda.out‚1293.4775390625 disk.nvme1n1.in‚ 744.57421875 disk.nvme1n1.out‚ 50406.53125 disk.md0.out‚ 1244.171875 disk.sdb2.in‚3.78125 disk.rbd0.out‚ 3.025390625 disk.md0p1.out‚ 1244.171875 disk.sdb.out‚1293.4775390625 disk.md0.in‚ 12.296875 proc.cpu.threads‚17 network.sent‚ 2.5871628e+07% disk./.usageGB‚2710.0922508239746! disk.sdb2.out‚1293.4775390625! disk.dm-0.out‚167103.30859375 disk.sda2.in‚8.515625 disk.rbd2.in‚0$ memory_percent‚92.03653335571289š$b’Þ(‚(:‡ ‘ëéÔײm gpu.0.gpu‚55.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.9915 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£»4ß(/ _wandbruntime‚11607  _runtime‚11607² ±ïÈà( trainer/global_step‚9749 % _timestamp‚1.7885075499130495e+09 _runtime‚11607.690306927 _step‚800 & lr-AdamW/pg1‚0.0001156830369387619 & lr-AdamW/pg2‚0.0001156830369387619÷J›îÍá(Ç & lr-AdamW/pg1‚0.0001156830369387619 & lr-AdamW/pg2‚0.0001156830369387619 trainer/global_step‚9749 % _timestamp‚1.7885075499130495e+09 _runtime‚11607.690306927 _step‚800 . }œ4â(/ _wandbruntime‚11607  _runtime‚11607ÄëÚ‡(ã(P P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 451600163 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚795858138432560 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 trainer/global_step‚9749 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 247229465 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11141892779404 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11184611715584 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11675626835456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚823157440718336 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 181740235 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚823157422715392 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 181739605 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 204370696 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚834342052433920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚919756082752512 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚834341124459520 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 204370696 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚801 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚823157422715392 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 247203121 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 247229465 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚823157440718336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11183701744128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 204370698 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚630 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59279559 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚807000956226328 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚807000031211964 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725è¢ Ø‹ä(†P LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚919756082752512 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚834341124459520 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 204370696 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚801 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚823157422715392 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 247203121 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 247229465 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚823157440718336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11183701744128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 204370698 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚630 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59279559 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚807000956226328 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚807000031211964 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 451600163 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚795858138432560 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 trainer/global_step‚9749 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 247229465 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11141892779404 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11184611715584 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11675626835456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚823157440718336 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 181740235 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚823157422715392 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 181739605 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 204370696 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚834342052433920¡«Iü[4å(/ _wandbruntime‚11608  _runtime‚11608–´t—'æ(ŠN H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚9749 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 204390858 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 122472111 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 451645180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11185733189120 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚11608.658787126 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59285945 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11143010052682 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 451645180 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11676791423488 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 451618808 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 247254322 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 247227952 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚802 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚834424792993280 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚834423865005568 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚908170681621504 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11677020192256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚807081091094358 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59285312 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚919847473044992 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚823239059804160 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 181757423 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚823239041801216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11143918027894 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 451618808 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚795937156027104 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚795937173066464 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 122472113 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚807080166079786 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚635 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚908170705561600 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 204390858 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 204390856 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚834424792993280 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 204390856 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 247227952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11184823204352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 247254322 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615ÄP¤˜'ç(N MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚9749 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 204390858 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 122472111 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 451645180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11185733189120 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚11608.658787126 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59285945 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11143010052682 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 451645180 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11676791423488 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 451618808 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 247254322 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 247227952 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚802 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚834424792993280 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚834423865005568 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚908170681621504 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11677020192256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚807081091094358 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59285312 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚919847473044992 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚823239059804160 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 181757423 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚823239041801216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11143918027894 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 451618808 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚795937156027104 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚795937173066464 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 122472113 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚807080166079786 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚635 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚908170705561600 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 204390858 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 204390856 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚834424792993280 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 204390856 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 247227952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11184823204352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 247254322¢+û™è4è(/ _wandbruntime‚11608  _runtime‚11608üÇ2aé(Û 1loss/SRqqt1/detection_loss‚2.2120283720669924 epoch‚8 0loss/FRt1/assignment_loss‚2.3848194862023377 2loss/SRqqt1/assignment_loss‚1.1719453995702442 2loss/SRqqt3/assignment_loss‚0.9621382050160596 2loss/SRqqt4/assignment_loss‚1.6446929390177778 /loss/FBt4/detection_loss‚2.2683350128286026 %loss/total_loss‚39.15869284527998 _step‚803 1loss/FBt3/assignment_loss‚0.08675909523541729 1loss/SRqqt4/detection_loss‚2.3996909688348356 % _timestamp‚1.7885075508828473e+09 _runtime‚11608.659836522 0loss/FRt4/assignment_loss‚2.7642915558815004 1loss/FBt1/assignment_loss‚0.04580293112752203 1loss/SRqqt3/detection_loss‚0.7279684278391191 /loss/FBt1/detection_loss‚0.6130234170566077 /loss/FRt3/detection_loss‚1.5206014101346166 .loss/FBt2/detection_loss‚2.707007849216461 1loss/FBt2/assignment_loss‚0.13782592041334282 0loss/SRqqt2/detection_loss‚2.066327948655401 /loss/FBt3/detection_loss‚2.6314685543378196 0loss/FRt2/assignment_loss‚2.7923040299311928 .loss/FRt4/detection_loss‚2.916545083522797 trainer/global_step‚9749 0loss/FRt3/assignment_loss‚1.1763606940285634 2loss/SRqqt2/assignment_loss‚1.5832876674830914 1loss/FBt4/assignment_loss‚0.24735971667900336 /loss/FRt1/detection_loss‚1.9513042598846368 /loss/FRt2/detection_loss‚2.1468039003160335¥Wâ¸fê(à 1loss/FBt4/assignment_loss‚0.24735971667900336 /loss/FRt1/detection_loss‚1.9513042598846368 /loss/FRt2/detection_loss‚2.1468039003160335 1loss/SRqqt1/detection_loss‚2.2120283720669924 epoch‚8 0loss/FRt1/assignment_loss‚2.3848194862023377 2loss/SRqqt1/assignment_loss‚1.1719453995702442 2loss/SRqqt3/assignment_loss‚0.9621382050160596 2loss/SRqqt4/assignment_loss‚1.6446929390177778 /loss/FBt4/detection_loss‚2.2683350128286026 %loss/total_loss‚39.15869284527998 _step‚803 1loss/FBt3/assignment_loss‚0.08675909523541729 1loss/SRqqt4/detection_loss‚2.3996909688348356 % _timestamp‚1.7885075508828473e+09 _runtime‚11608.659836522 0loss/FRt4/assignment_loss‚2.7642915558815004 1loss/FBt1/assignment_loss‚0.04580293112752203 1loss/SRqqt3/detection_loss‚0.7279684278391191 /loss/FBt1/detection_loss‚0.6130234170566077 /loss/FRt3/detection_loss‚1.5206014101346166 .loss/FBt2/detection_loss‚2.707007849216461 1loss/FBt2/assignment_loss‚0.13782592041334282 0loss/SRqqt2/detection_loss‚2.066327948655401 /loss/FBt3/detection_loss‚2.6314685543378196 0loss/FRt2/assignment_loss‚2.7923040299311928 .loss/FRt4/detection_loss‚2.916545083522797 trainer/global_step‚9749 0loss/FRt3/assignment_loss‚1.1763606940285634 2loss/SRqqt2/assignment_loss‚1.5832876674830914£ •ªXdë(‚(:Ù  ëéÔù‡Äm disk.md0p1.out‚ 1245.91796875) proc.memory.availableMB‚ 1303.83203125% disk./.usageGB‚2709.5646743774414 disk.rbd0.in‚0 disk.dm-0.in‚ 1684.23828125 disk.sda2.in‚8.515625$ memory_percent‚92.04204082489014 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sdb2.in‚3.78125 disk.sda.in‚8.515625 disk.rbd0.out‚ 3.025390625 disk.nvme1n1.out‚ 50406.8125 cpu‚25.21228542598727 disk.sda1.out‚0 disk.nvme0n1.in‚ 939.6640625 disk.nvme1n1.in‚ 744.57421875 disk.md0.in‚ 12.296875 disk.md0.out‚ 1245.91796875" disk.sdb2.out‚1295.28662109375 disk.sda1.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚ 12.296875 disk.dm-0.out‚ 167152.3125 disk.sdb.in‚3.78125 disk.nvme0n1.out‚116745.5) disk./.usagePercent‚75.76056233651074! disk.sdb.out‚1295.28662109375* proc.memory.percent‚18.393707275390625! disk.sda.out‚1295.28662109375" disk.sda2.out‚1295.28662109375 proc.memory.rssMB‚3013.625 network.recv‚ 3.779123e+06 network.sent‚ 2.5888271e+07 disk.rbd2.out‚ 3.537109375 disk.sdb1.in‚0×Ö¾‘“ì(‚(:ˆ  ëéÔä•ë| gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚124.321' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.08025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢ß.œîí(‚(:ˆ ¯ëéÔ€÷‡m gpu.0.gpu‚31.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚176.461' gpu.0.enforcedPowerLimitWattsðtw©¥‚400.0 gpu.0.powerPercent‚44.11525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å >Ójî(‚(:ß ¯ëéÔÔÏm disk.rbd0.out‚ 3.025390625 disk.nvme0n1.in‚ 939.6640625" disk.sdb2.out‚1296.86962890625 disk.sda.in‚8.515625 disk.sda1.in‚0 disk.dm-0.out‚ 167154.34375" disk.sda2.out‚1296.86962890625 disk.nvme1n1.in‚ 744.57421875 cpu‚25.212013948628602 network.recv‚ 3.784474e+06 disk.sda1.out‚0 disk.sdb1.in‚0# disk.nvme0n1.out‚116747.2265625 proc.memory.rssMB‚3013.625* proc.memory.percent‚18.393707275390625 disk.sdb2.in‚3.78125 disk.dm-0.in‚ 1684.23828125$ memory_percent‚92.04111099243164 disk.md0.in‚ 12.296875 disk.md0.out‚ 1247.45703125 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚ 12.296875 disk.sdb.in‚3.78125! disk.sdb.out‚1296.86962890625 disk.sda2.in‚8.515625! disk.sda.out‚1296.86962890625" disk.nvme1n1.out‚ 50407.1171875 network.sent‚ 2.59852e+07 disk.rbd2.in‚0' proc.memory.availableMB‚ 1303.984375) disk./.usagePercent‚75.75639809601564$ disk./.usageGB‚2709.415740966797 disk.md0p1.out‚ 1247.45703125 disk.rbd2.out‚ 3.537109375 proc.cpu.threads‚17-½;SYï(‚(:Î ¾ëéÔÎí¦m disk.sdb1.out‚0 disk.rbd2.out‚ 3.537109375 disk.sdb2.in‚3.78125' proc.memory.availableMB‚ 1303.671875 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.md0p1.out‚ 1248.80859375 disk.nvme1n1.in‚ 744.57421875 disk.sdb2.out‚ 1298.26953125* proc.memory.percent‚18.393707275390625 network.recv‚ 3.789184e+06 disk.sda.in‚8.515625 disk.sda1.in‚0 disk.sdb.in‚3.78125 disk.rbd2.in‚0 disk.sda1.out‚0 cpu‚25.21178338730017 proc.cpu.threads‚17 disk.rbd0.out‚ 3.025390625 disk.sda2.in‚8.515625$ memory_percent‚92.04301834106445( disk./.usagePercent‚75.7606302792697 disk.nvme0n1.in‚ 939.6640625 disk.md0.out‚ 1248.80859375 network.sent‚ 2.5991483e+07% disk./.usageGB‚2709.5671043395996 disk.sda.out‚ 1298.26953125 disk.md0p1.in‚ 12.296875 disk.dm-0.out‚ 167310.140625 disk.nvme0n1.out‚116747.5 disk.dm-0.in‚ 1684.23828125 disk.sdb.out‚ 1298.26953125 disk.sda2.out‚ 1298.26953125! disk.nvme1n1.out‚ 50562.640625 disk.md0.in‚ 12.296875 proc.memory.rssMB‚3013.625[ [¬“ð(‚(:ˆ ¾ëéÔÓ¸m gpu.0.gpu‚11.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚123.435' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.85875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛÓ+M“ñ(‚(:ˆ ÍëéÔÚ¾øl gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.989' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.24725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï0‚H[ò(‚(:Ð ÍëéÔ›çƒn disk.rbd0.out‚ 3.025390625 disk.sdb.in‚3.78125 cpu‚25.21155094911552( disk./.usagePercent‚75.7637054090713 disk.sdb2.in‚3.78125 disk.sda.in‚8.515625 disk.sdb1.in‚0 disk.dm-0.out‚ 167390.875" disk.sda2.out‚1299.94091796875( proc.memory.availableMB‚ 1303.9140625 proc.cpu.threads‚17$ disk./.usageGB‚2709.677085876465" disk.sdb2.out‚1299.94091796875 disk.md0p1.in‚ 12.296875 disk.nvme0n1.in‚ 939.6640625 proc.memory.rssMB‚3013.625 network.sent‚ 2.5995809e+07 disk.nvme1n1.in‚ 744.57421875! disk.sda.out‚1299.94091796875 disk.sda1.out‚0 disk.md0.in‚ 12.296875 disk.sda1.in‚0 disk.dm-0.in‚ 1684.23828125! disk.sdb.out‚1299.94091796875* proc.memory.percent‚18.393707275390625 disk.nvme0n1.out‚116747.5 disk.nvme1n1.out‚ 50643.375 disk.sdb1.out‚0 disk.md0p1.out‚ 1250.421875 disk.rbd2.in‚0 disk.rbd2.out‚ 3.537109375 disk.rbd0.in‚0 disk.sda2.in‚8.515625$ memory_percent‚92.04154014587402 network.recv‚ 3.792161e+06 disk.md0.out‚ 1250.421875âĈr4ó(/ _wandbruntime‚11663  _runtime‚11663TwÆHÉô(à _runtime‚11663.16732999 _step‚804 ' lr-AdamW/pg1‚0.00011097394294355735 ' lr-AdamW/pg2‚0.00011097394294355735 trainer/global_step‚9799 % _timestamp‚1.7885076053898702e+09åT3Îõ(È _step‚804 ' lr-AdamW/pg1‚0.00011097394294355735 ' lr-AdamW/pg2‚0.00011097394294355735 trainer/global_step‚9799 % _timestamp‚1.7885076053898702e+09 _runtime‚11663.16732999¤¿õ¯&4ö(/ _wandbruntime‚11663  _runtime‚11663ÐêW(÷(P TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚912591765079552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚912591741139456 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚827238377006592 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚924325827910656 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 205378696 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 241364992 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11239789169152 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚838478166175744 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 123063296 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11733821466112 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚630 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59567202 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 182630500 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚838478166175744 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 248474367 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59566572 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 265305088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 248448023 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚827238395009536 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11240699140608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 205378696 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 453826719 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚811006788194272 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚11663.168446675 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11734062831104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11734062831104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚912591741139456 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚827238377006592 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚924325827910656 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 205378696 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 241364992 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11239789169152 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚838478166175744 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 123063296 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11733821466112 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚630 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59567202 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 182630500 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚838478166175744 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 248474367 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59566572 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 265305088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 248448023 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚827238395009536 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11240699140608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 205378696 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 453826719 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚811006788194272 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚11663.168446675 Joú(‹N OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 248472854 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 123075363 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 453898082 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 182647540 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11241820614144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚827320014095360 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 123075361 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚827319996092416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11198887307790 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 453871710 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚827320014095360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚811086923062094 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 205398856 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 453898082 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 248472854 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 182648160 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚827319996092416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59572179 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚924416959801856 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚912681990784000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 248499224 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 248499224 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚924417210413568 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚799888052793664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11735219629568 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 205398856 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚799888035754304 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 248472854 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 182648160 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚827319996092416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59572179 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚924416959801856 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚912681990784000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 248499224 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 248499224 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚924417210413568 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚799888052793664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11735219629568 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 205398856 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚799888035754304 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 182647540 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11241820614144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚827320014095360 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 123075361 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚827319996092416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11198887307790 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 453871710 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚827320014095360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚811086923062094 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 205398856 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 453898082 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185¦þ„by4ü(/ _wandbruntime‚11664  _runtime‚11664m‡xÏYý(Ó /loss/FRt4/detection_loss‚3.1135881922461768 1loss/SRqqt3/detection_loss‚0.7937981652203894 .loss/FBt2/detection_loss‚2.354782142241796 % _timestamp‚1.7885076064311533e+09 /loss/FRt1/detection_loss‚1.9900127171761994 1loss/SRqqt2/detection_loss‚2.6987108039855956 /loss/FBt1/detection_loss‚0.7068236527060542 .loss/FBt3/detection_loss‚2.658406933148702 _runtime‚11664.208359275 _step‚807 0loss/FRt3/assignment_loss‚1.0351187813849676 1loss/SRqqt2/assignment_loss‚1.201798377931118 0loss/FBt3/assignment_loss‚0.2333936578943394 /loss/FRt3/detection_loss‚1.5267760658429728 /loss/FBt4/detection_loss‚2.0884141193495855 trainer/global_step‚9799 2loss/SRqqt1/assignment_loss‚1.3456691894680262 1loss/SRqqt4/assignment_loss‚1.545208325489394 -loss/FRt2/detection_loss‚2.79104763378614 0loss/SRqqt4/detection_loss‚2.457246741940898 /loss/FRt1/assignment_loss‚2.252456176010045 2loss/SRqqt3/assignment_loss‚0.8016724855288108 1loss/FBt2/assignment_loss‚0.20405740561133523 0loss/FBt4/assignment_loss‚0.2505222131635871 1loss/FBt1/assignment_loss‚0.14948681725628887 1loss/SRqqt1/detection_loss‚1.8912642553448678 epoch‚8 0loss/FRt4/assignment_loss‚2.4488403363661333 %loss/total_loss‚40.55859409637997 /loss/FRt2/assignment_loss‚4.019498907286545ë ÞÊ^þ(Ø %loss/total_loss‚40.55859409637997 /loss/FRt2/assignment_loss‚4.019498907286545 /loss/FRt4/detection_loss‚3.1135881922461768 1loss/SRqqt3/detection_loss‚0.7937981652203894 .loss/FBt2/detection_loss‚2.354782142241796 % _timestamp‚1.7885076064311533e+09 /loss/FRt1/detection_loss‚1.9900127171761994 1loss/SRqqt2/detection_loss‚2.6987108039855956 /loss/FBt1/detection_loss‚0.7068236527060542 .loss/FBt3/detection_loss‚2.658406933148702 _runtime‚11664.208359275 _step‚807 0loss/FRt3/assignment_loss‚1.0351187813849676 1loss/SRqqt2/assignment_loss‚1.201798377931118 0loss/FBt3/assignment_loss‚0.2333936578943394 /loss/FRt3/detection_loss‚1.5267760658429728 /loss/FBt4/detection_loss‚2.0884141193495855 trainer/global_step‚9799 2loss/SRqqt1/assignment_loss‚1.3456691894680262 1loss/SRqqt4/assignment_loss‚1.545208325489394 -loss/FRt2/detection_loss‚2.79104763378614 0loss/SRqqt4/detection_loss‚2.457246741940898 /loss/FRt1/assignment_loss‚2.252456176010045 2loss/SRqqt3/assignment_loss‚0.8016724855288108 1loss/FBt2/assignment_loss‚0.20405740561133523 0loss/FBt4/assignment_loss‚0.2505222131635871 1loss/FBt1/assignment_loss‚0.14948681725628887 1loss/SRqqt1/detection_loss‚1.8912642553448678 epoch‚8 0loss/FRt4/assignment_loss‚2.4488403363661333§²½aœÿ(‚(:‘ ÜëéÔ¦ÝÑm gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚119.978' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.994499999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BŸù@o€)‚(:ä ÜëéÔŽ¯n" disk.sda2.out‚1301.74951171875) proc.memory.availableMB‚ 1303.84765625# disk.nvme1n1.out‚50816.92578125 network.recv‚ 3.79667e+06 disk.sda1.out‚0 disk.md0p1.in‚ 12.296875 disk.nvme1n1.in‚ 744.57421875) disk./.usagePercent‚75.76994942328224 disk.rbd2.in‚0" disk.sdb2.out‚1301.74951171875! disk.sda.out‚1301.74951171875 disk.sdb1.out‚0! disk.dm-0.out‚167564.66015625 disk.sda.in‚8.515625 disk.dm-0.in‚ 1684.23828125 disk.sda2.in‚8.515625 disk.md0.in‚ 12.296875# memory_percent‚92.0419454574585 proc.cpu.threads‚17 network.sent‚ 2.5999584e+07 disk.sdb2.in‚3.78125 disk.sdb1.in‚0 cpu‚25.211256739956085$ disk./.usageGB‚2709.900402069092 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 3.025390625 disk.md0p1.out‚ 1252.17578125 disk.md0.out‚ 1252.17578125 proc.memory.rssMB‚3013.625! disk.sdb.out‚1301.74951171875* proc.memory.percent‚18.393707275390625 disk.rbd2.out‚ 3.537109375 disk.sdb.in‚3.78125 disk.nvme0n1.in‚ 939.6640625" disk.nvme0n1.out‚ 116747.734375¨ ¬g“)‚(:ˆ ëëéÔüÄÛk gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸–¢m‚)‚(:â ëëéÔìÄÜl disk.sdb.in‚3.78125 disk.nvme0n1.in‚ 939.6640625 disk.sdb1.out‚0" disk.nvme0n1.out‚ 116747.734375 disk.md0.in‚ 12.296875" disk.sdb2.out‚1303.57958984375$ memory_percent‚92.04208850860596! disk.sdb.out‚1303.57958984375" disk.sda2.out‚1303.57958984375 disk.nvme1n1.in‚ 744.57421875 disk.rbd2.in‚0 disk.sdb1.in‚0) proc.memory.availableMB‚ 1303.82421875 cpu‚25.210950806779227 network.recv‚ 3.802748e+06% disk./.usageGB‚2710.1503677368164 disk.dm-0.in‚ 1684.23828125 disk.sda.in‚8.515625 disk.md0p1.in‚ 12.296875 disk.dm-0.out‚ 167685.03125! disk.nvme1n1.out‚ 50937.296875 disk.sdb2.in‚3.78125 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.out‚ 1253.94140625 disk.md0.out‚ 1253.94140625 disk.rbd2.out‚ 3.537109375 disk.rbd0.out‚ 3.025390625* proc.memory.percent‚18.393707275390625 network.sent‚ 2.6107459e+07) disk./.usagePercent‚75.77693856797057 disk.sda2.in‚8.515625! disk.sda.out‚1303.57958984375 disk.rbd0.in‚0 proc.memory.rssMB‚3013.625 proc.cpu.threads‚17“ƒ)‚(:ˆ úëéÔñˆæk gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³Õdn„)‚(:ã úëéÔ®Ñäl disk.sda.in‚8.515625 disk.md0p1.out‚ 1255.6171875 disk.sdb.in‚3.78125( proc.memory.availableMB‚ 1303.3359375 network.recv‚ 3.807604e+06 disk.sda1.out‚0 disk.dm-0.in‚ 1684.23828125 disk.nvme0n1.in‚ 939.6640625 disk.nvme1n1.in‚ 744.57421875) disk./.usagePercent‚75.77868748071577$ disk./.usageGB‚2710.212917327881 disk.rbd0.in‚0! disk.sdb.out‚1305.32763671875 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.md0p1.in‚ 12.296875" disk.sda2.out‚1305.32763671875$ memory_percent‚92.04506874084473 disk.sdb2.in‚3.78125" disk.sdb2.out‚1305.32763671875 disk.dm-0.out‚ 167916.171875 disk.md0.in‚ 12.296875 proc.memory.rssMB‚3013.625! disk.sda.out‚1305.32763671875 disk.sda1.in‚0 disk.sda2.in‚8.515625$ disk.nvme0n1.out‚116747.98046875# disk.nvme1n1.out‚51168.19140625 disk.rbd2.in‚0* proc.memory.percent‚18.393707275390625 cpu‚25.210616706484537 network.sent‚ 2.6113855e+07 disk.rbd0.out‚ 3.025390625 disk.sdb1.out‚0 disk.md0.out‚ 1255.6171875 disk.rbd2.out‚ 3.537109375š}*G’…)‚(:‡ ‰ìéÔ¶Ö÷k gpu.0.gpu‚45.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.774' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.9435 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µâ©ðq†)‚(:æ ‰ìéÔÏöìl disk.rbd2.in‚0! disk.sdb.out‚1307.26318359375( proc.memory.availableMB‚ 1305.4921875$ disk./.usageGB‚2710.275417327881 disk.sda1.out‚0 disk.sdb1.in‚0 disk.dm-0.in‚ 1684.23828125$ memory_percent‚92.03190803527832 network.sent‚ 2.6119157e+07) disk./.usagePercent‚75.78043500687406 disk.sdb2.in‚3.78125! disk.sda.out‚1307.26318359375 disk.sda1.in‚0 disk.rbd0.out‚ 3.025390625 disk.sda.in‚8.515625 disk.md0p1.in‚ 12.296875 disk.sdb.in‚3.78125 disk.nvme1n1.in‚ 744.57421875 disk.md0.in‚ 12.296875 disk.md0.out‚ 1257.48828125 disk.rbd2.out‚ 3.537109375" disk.sdb2.out‚1307.26318359375 disk.sda2.in‚8.515625# disk.nvme1n1.out‚51289.69921875 disk.rbd0.in‚0 disk.dm-0.out‚168037.6796875" disk.sda2.out‚1307.26318359375 network.recv‚ 3.811015e+06 disk.sdb1.out‚0$ disk.nvme0n1.out‚116747.98046875 proc.memory.rssMB‚3013.625* proc.memory.percent‚18.393707275390625 disk.md0p1.out‚ 1257.48828125 disk.nvme0n1.in‚ 939.6640625 cpu‚25.210304735472665 proc.cpu.threads‚17òA L4‡)/ _wandbruntime‚11716  _runtime‚11716ìKêIʈ)Ä ' lr-AdamW/pg1‚0.00010635137661554879 ' lr-AdamW/pg2‚0.00010635137661554879 trainer/global_step‚9849 % _timestamp‚1.7885076587559228e+09 _runtime‚11716.533315608 _step‚808 ÌEsω)É _runtime‚11716.533315608 _step‚808 ' lr-AdamW/pg1‚0.00010635137661554879 ' lr-AdamW/pg2‚0.00010635137661554879 trainer/global_step‚9849 % _timestamp‚1.7885076587559228e+09¨róiÿ4Š)/ _wandbruntime‚11716  _runtime‚11716Øeo(‹)€P ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚831319349300736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚815014470190944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚815013545176580 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚917103026361856 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 456079621 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 249719269 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 249719269 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59853216 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 206386696 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 123666548 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚831319331297792 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚842616135866368 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11792205718016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚803759914926320 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚842615207891968 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11296786565632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0àhúÂŒ)…P XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59853216 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 M?DeviceStatsQvÿžI'Monitor.on_train_batch_start/active.large_pool.freed‚ 206386696 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 123666548 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚831319331297792 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚842616135866368 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11792205718016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚803759914926320 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚842615207891968 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11296786565632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚831319349300736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚815014470190944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚815013545176580 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚917103026361856 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 456079621 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 249719269 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 249719269©@Ÿ 4)/ _wandbruntime‚11717  _runtime‚11717ý âr'Ž)‹N SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 249744126 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚815094605058974 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚917193276006400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚831400968386560 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 456124612 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11296998054400 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 206406858 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 206406858 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚842697948438016 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 456150984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11296998054400 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 249744126 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 183536901 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚842698876425728 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 183537523 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚803838932520864 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚831400950383616 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 249717756 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59858290 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11255672538110 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 249717756 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 267388928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚815093680044402 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 % _timestamp‚1.7885076598457153e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11297908039168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚928986871728640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 243448832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 123678611 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 456124612 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11296998054400 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 206406858 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 206406858 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚842697948438016 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 456150984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11296998054400 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 249744126 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 183536901 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚842698876425728 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 183537523 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚803838932520864 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚831400950383616 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 249717756 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59858290 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11255672538110 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 249717756 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 267388928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚815093680044402 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 % _timestamp‚1.7885076598457153e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11297908039168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚928986871728640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 243448832ª;#ª4)/ _wandbruntime‚11717  _runtime‚11717ès¿]‘)× 2loss/SRqqt2/assignment_loss‚1.4440234030286472 /loss/FBt3/detection_loss‚2.4998965859413147 %loss/total_loss‚39.05435327828047 0loss/FRt1/assignment_loss‚1.5657324177878245 1loss/SRqqt1/assignment_loss‚1.319808245110123 1loss/FBt2/assignment_loss‚0.11953351663339618 .loss/FRt2/detection_loss‚2.291333873391799 0loss/FBt1/assignment_loss‚0.0979676598192041 1loss/FBt4/assignment_loss‚0.18162326842223137 /loss/FRt1/detection_loss‚2.0809677153647237 .loss/FRt3/detection_loss‚1.765503539525448 1loss/SRqqt1/detection_loss‚2.0869170051553976 0loss/FRt3/assignment_loss‚1.2496281266212463 /loss/FRt4/assignment_loss‚3.057155029340224 2loss/SRqqt3/assignment_loss‚1.0180257291330592 _runtime‚11717.626214727 2loss/SRqqt4/assignment_loss‚1.1940410069443963 1loss/FBt3/assignment_loss‚0.12801148794304545 0loss/SRqqt3/detection_loss‚1.005119090332933 epoch‚8 % _timestamp‚1.7885076598491726e+09 0loss/FRt2/assignment_loss‚2.6162196579186814 1loss/SRqqt2/detection_loss‚2.3728175520896913 0loss/SRqqt4/detection_loss‚2.503522065552798 trainer/global_step‚9849 _step‚811 /loss/FBt1/detection_loss‚0.6599303883073541 /loss/FRt4/detection_loss‚3.2253010164607656 /loss/FBt2/detection_loss‚1.8363834768533707 /loss/FBt4/detection_loss‚2.7348914206027986BIîš’)Ü 0loss/FRt2/assignment_loss‚2.6162196579186814 1loss/SRqqt2/detection_loss‚2.3728175520896913 0loss/SRqqt4/detection_loss‚2.503522065552798 trainer/global_step‚9849 _step‚811 /loss/FBt1/detection_loss‚0.6599303883073541 /loss/FRt4/detection_loss‚3.2253010164607656 /loss/FBt2/detection_loss‚1.8363834768533707 /loss/FBt4/detection_loss‚2.7348914206027986 2loss/SRqqt2/assignment_loss‚1.4440234030286472 /loss/FBt3/detection_loss‚2.4998965859413147 %loss/total_loss‚39.05435327828047 0loss/FRt1/assignment_loss‚1.5657324177878245 1loss/SRqqt1/assignment_loss‚1.319808245110123 1loss/FBt2/assignment_loss‚0.11953351663339618 .loss/FRt2/detection_loss‚2.291333873391799 0loss/FBt1/assignment_loss‚0.0979676598192041 1loss/FBt4/assignment_loss‚0.18162326842223137 /loss/FRt1/detection_loss‚2.0809677153647237 .loss/FRt3/detection_loss‚1.765503539525448 1loss/SRqqt1/detection_loss‚2.0869170051553976 0loss/FRt3/assignment_loss‚1.2496281266212463 /loss/FRt4/assignment_loss‚3.057155029340224 2loss/SRqqt3/assignment_loss‚1.0180257291330592 _runtime‚11717.626214727 2loss/S ¸0|ÈRqqt4/assignment_loss‚1.1940410069443963 1loss/FBt3/assignment_loss‚0.12801148794304545 0loss/SRqqt3/detection_loss‚1.005119090332933 epoch‚8 % _timestamp‚1.7885076598491726e+09«žAd‘“)‚(:† ˜ìéÔ˜—¸k gpu.0.gpu‚41.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.168' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 8oØe”)‚(:Ú ˜ìéÔ¼±Ÿm disk.md0.in‚ 12.296875 disk.sda.in‚8.515625$ memory_percent‚92.03269481658936 network.sent‚ 2.6125097e+07 disk.dm-0.in‚ 1684.23828125$ disk.nvme0n1.out‚116748.07421875# disk.nvme1n1.out‚51290.17578125 disk.rbd2.in‚0 disk.sdb2.in‚3.78125 disk.sda1.out‚0 disk.rbd0.in‚0 network.recv‚ 3.815594e+06 disk.dm-0.out‚ 168038.25 disk.sdb2.out‚1308.830078125 disk.sdb1.in‚0 disk.md0p1.out‚ 1258.98828125 cpu‚25.210055939045823 proc.cpu.threads‚17 disk.sdb.out‚1308.830078125) proc.memory.availableMB‚ 1305.36328125 disk.sdb.in‚3.78125 disk.nvme0n1.in‚ 939.6640625 disk.rbd2.out‚ 3.537109375 disk.sda1.in‚0 disk.rbd0.out‚ 3.025390625( disk./.usagePercent‚75.7692008796678 disk.sdb1.out‚0 disk.md0p1.in‚ 12.296875* proc.memory.percent‚18.393707275390625 disk.nvme1n1.in‚ 744.57421875 disk.md0.out‚ 1258.98828125 disk.sda.out‚1308.830078125% disk./.usageGB‚2709.8736305236816 disk.sda2.in‚8.515625 disk.sda2.out‚1308.830078125 proc.memory.rssMB‚3013.625êƒ|¦“•)‚(:ˆ §ìéÔ†æk gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~·Ápn–)‚(:ã §ìéÔ¢·±m! disk.sdb.out‚1310.27978515625 disk.md0.in‚ 12.296875! disk.sda.out‚1310.27978515625# memory_percent‚92.0316219329834 proc.memory.rssMB‚3013.625$ disk.nvme0n1.out‚116748.08203125" disk.sdb2.out‚1310.27978515625 disk.sda.in‚8.515625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb1.in‚0 disk.dm-0.in‚ 1684.23828125 disk.dm-0.out‚168259.6171875 disk.nvme0n1.in‚ 939.6640625 disk.sdb1.out‚0 disk.md0p1.in‚ 12.296875" disk.sda2.out‚1310.27978515625 disk.nvme1n1.in‚ 744.57421875# disk.nvme1n1.out‚51511.53515625 disk.sdb.in‚3.78125 disk.sdb2.in‚3.78125 disk.rbd0.out‚ 3.025390625 network.sent‚ 2.6234901e+07 disk.sda2.in‚8.515625 disk.rbd2.out‚ 3.537109375* proc.memory.percent‚18.393707275390625 cpu‚25.209808802042705 proc.cpu.threads‚17 network.recv‚ 3.820986e+06) disk./.usagePercent‚75.77477613234234 disk.rbd0.in‚0( proc.memory.availableMB‚ 1305.5390625$ disk./.usageGB‚2710.073028564453 disk.md0p1.out‚ 1260.3828125 disk.md0.out‚ 1260.3828125 disk.rbd2.in‚0.–m“—)‚(:ˆ ¶ìéÔ©Œàk gpu.0.gpu‚32.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ò£1o˜)‚(:ä ¶ìéÔÄým disk.dm-0.out‚168494.4296875( proc.memory.availableMB‚ 1305.5390625 proc.memory.rssMB‚3013.625 disk.sdb.in‚3.78125 network.sent‚ 2.6241327e+07 disk.rbd0.out‚ 3.025390625 disk.sdb1.out‚0 disk.md0p1.out‚ 1262.3046875$ disk.nvme0n1.out‚116748.30078125 disk.md0.in‚ 12.296875 disk.md0.out‚ 1262.3046875 disk.rbd2.in‚0 disk.rbd2.out‚ 3.537109375 disk.sda1.in‚0 cpu‚25.209521782037847 disk.nvme0n1.in‚ 939.6640625 disk.nvme1n1.in‚ 744.57421875# disk.nvme1n1.out‚51746.12890625 network.recv‚ 3.825909e+06 disk.sda2.in‚8.515625! disk.sda.out‚1312.26708984375! disk.sdb.out‚1312.26708984375% disk./.usageGB‚2710.1337814331055 disk.rbd0.in‚0 disk.sdb1.in‚0" disk.sdb2.out‚1312.26708984375* proc.memory.percent‚18.393707275390625 proc.cpu.threads‚17 disk.sdb2.in‚3.78125 disk.sda.in‚8.515625 disk.sda1.out‚0 disk.dm-0.in‚ 1684.23828125) disk./.usagePercent‚75.77647480797691" disk.sda2.out‚1312.26708984375# memory_percent‚92.0316219329834 disk.md0p1.in‚ 12.296875ÑnM¶4™)/ _wandbruntime‚11773  _runtime‚11773PS¥QÊš)Ä % _timestamp‚1.7885077155598452e+09 _runtime‚11773.337200817 _step‚812 ' lr-AdamW/pg1‚0.00010181630129063056 ' lr-AdamW/pg2‚0.00010181630129063056 trainer/global_step‚9899‹O³ÄÏ›)É trainer/global_step‚9899 % _timestamp‚1.7885077155598452e+09 _runtime‚11773.337200817 _step‚812 ' lr-AdamW/pg1‚0.00010181630129063056 ' lr-AdamW/pg2‚0.00010181630129063056¬Œ_jâ4œ)/ _wandbruntime‚11773  _runtime‚11773²Ûø()P VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240139264 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚846753177582592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚846752249608192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚11773.338555563 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚835400285588992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11352873990656 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 184409213 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 % _timestamp‚1.7885077155608935e+09 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 207394696 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚835400303591936 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11352873990656 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11351964019200 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚846752249608192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 458358869 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11850582877696 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 250964171 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚813 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11309524544728 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚819021227173252 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 250937827 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 124269796 trainer/global_step‚9899 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚615 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚921614311584256 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 207394698 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 124269798 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 207394698 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60140030 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚835400285588992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚807710794653520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 458332523 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11850799076864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚835400303591936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60139417 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚921614335524352 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 184409828 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 184409828 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚846752249608192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚11773.338555563 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚835400285588992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11352873990656 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 184409213 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 % _timestamp‚1.7885077155608935e+09 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 207394696 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚835400303591936 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11352873990656 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11351964019200 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚846752249608192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 458358869 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11850582877696 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 250964171 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚813 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11309524544728 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚819021227173252 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 250937827 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 124269796 trainer/global_step‚9899 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚615 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚921614311584256 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 207394698 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 124269798 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 207394698 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60140030 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚835400285588992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0­ƒ^Û4Ÿ)/ _wandbruntime‚11774  _runtime‚11774‘Ž )ŠN YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11851980724736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11310641818006 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚835481904674816 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11353995464192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 207414856 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚807789795208704 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚819100437026710 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚846835918141952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 458403886 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚933556541953536 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚933556289244672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11353085479424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 458377514 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 184427015 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 124281863 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 250962658 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 250989028 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 458377514 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185]RÕò'¡)N QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚921704537288704 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11353995464192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚807789812248064 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 184427633 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11851751955968 % _timestamp‚1.7885077166801543e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60145770 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚835481904674816 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 250989028 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60145154 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 250962658 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11311549793218 _step‚814 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 184427015 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 124281863 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 250962658 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 250989028 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 458377514 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11851980724736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11310641818006 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚835481904674816 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11353995464192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 207414856 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚807789795208704 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚819100437026710 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚846835918141952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 458403886 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚933556541953536 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚933556289244672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11353085479424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 458377514 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60428610 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚823027059141196 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚811661657341360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚11830.274327193 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11408961415680 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 460611771 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 124873048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11909252155392 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11909025470464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚823027984155560 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 185301656 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚850890219298816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚850889291324416 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 252182729 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚839481239880192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADs&8ýÖeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1tUy. (²)†P SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11909252155392 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11909025470464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚823027984155560 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 185301656 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚850890219298816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚850889291324416 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 252182729 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚839481239880192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚850890219298816 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11408051444224 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60428610 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚823027059141196 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚811661657341360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚11830.274327193 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11408961415680 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 460611771 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 124873048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 460656788 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 185320052 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 460630416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚850972959858176 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 252207560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚811740691975264 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 124885111 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚811740674935904 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 trainer/global_step‚9949 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 208422858 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 252207560 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚839562858966016 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11409172904448 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11410082889216 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11367427048326 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚839562858966016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 208422856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 185319430 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚926215822511104 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚839562876968960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 _step‚818 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 208422856 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11910429984768 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚839562876968960 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 460630416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60434939 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚823107194009018 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 _runtime‚11831.299528678 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60434319 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚823108119023590 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11366519073114 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 252233930 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚850972031870464 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 $ _timestamp‚1.788507773521824e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 460656788 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0;¿då'µ)N MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11910207507456 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚850972959858176 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 460656788 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 185320052 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 460630416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚850972959858176 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 252207560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚811740691975264 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 124885111 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚811740674935904 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 trainer/global_step‚9949 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 208422858 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 252207560 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚839562858966016 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11409172904448 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11410082889216 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11367427048326 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚839562858966016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 208422856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 185319430 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚926215822511104 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚839562876968960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 _step‚818 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 208422856 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11910429984768 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚839562876968960 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 460630416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60434939 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚823107194009018 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 _runtime‚11831.299528678 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60434319 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚823108119023590 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11366519073114 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 252233930 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚850972031870464 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 $ _timestamp‚1.788507773521824e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 460656788 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚855026333040640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 186195301 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 209410698 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 125476298 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 209410696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11967510052864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11464138869248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60719003 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 186194656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 _step‚821 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11465048840704 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11465048840704 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚843562212174336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚843562194171392 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 trainer/global_step‚9999 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚843562194171392 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7885078293044548e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 209410698 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 125476298 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 209410696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11967510052864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11464138869248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60719003 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 186194656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 _step‚821 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11465048840704 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11465048840704 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚843562212174336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚843562194171392 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 trainer/global_step‚9999 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚843562194171392 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7885078293044548e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚855026333040640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 186195301 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 186212524 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚642 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 % _timestamp‚1.7885078302501614e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 253478832 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 462909690 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 462883318 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 253452462 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 209430856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 462883318 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11465260329472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11423304303434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚843643831260160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚827114876005898 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11422396328222 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚843643813257216 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚855109073586688 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚855110001574400 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 209430856 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 253478832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 186212524 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚642 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 % _timestamp‚1.7885078302501614e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 253478832 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 462909690 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 462883318 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 253452462 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 209430856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 462883318 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11465260329472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11423304303434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚843643831260160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚827114876005898 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11422396328222 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚843643813257216 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚855109073586688 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚855110001574400 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 209430856 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 253478832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚859163374756864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚831041498120176 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 465091229 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚819563416795760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61007815 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12026256757760 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚819563433835120 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11477156310052 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚831040573105812 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 465117575 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 187087361 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚859163374756864 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚847643166465536 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 254698877 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12026025878528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚847643166465536 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 254698877 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11520226294272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61008459 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11478064285056 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 _step‚825 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 126079548 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11521136265728 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚859164302731264 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 254672533 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚847643148462592 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.sma?´g‚ll_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚935148167251456 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615:›æ (Ø)‡P KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚859164302731264 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 126079546 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 187087361 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚859163374756864 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚847643166465536 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 254698877 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12026025878528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚847643166465536 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 254698877 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11520226294272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61008459 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11478064285056 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 _step‚825 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 126079548 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11521136265728 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚859164302731264 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 254672533 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚847643148462592 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚935148167251456 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 210418698 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 210418696 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚646 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚859163374756864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚831041498120176 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 465091229 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚819563416795760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61007815 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12026256757760 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚819563433835120 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11477156310052 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚831040573105812 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 465117575 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 J¹4Û)/ _wandbruntime‚11940  _runtime‚11940MÓ'Ü)ŒN T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚859247043290624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61014268 _runtime‚11940.641327366 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚847724767548416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚655 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 210438858 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 187105881 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 trainer/global_step‚10049 _step‚826 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61013613 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚947265842228224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 465162592 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11522257739264 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 126091611 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚947265589519360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12027425332224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 465136220 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 187105224 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚935238416896000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚819642434390304 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚935238392955904 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 254697364 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12027196563456 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 210438856 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚847724785551360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 126091613 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚831120707973634 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚859246115302912 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11479181558542 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚657 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 465162592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚831121632988206 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11478273583330 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚859247043290624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 210438856 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 210438858 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 254697364 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚847724767548416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚819642451429664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11521347754496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0iv‰'Ý)‘N TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 trainer/global_step‚10049 _step‚826 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61013613 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚947265842228224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 465162592 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11522257739264 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 126091611 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚947265589519360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12027425332224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 465136220 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 187105224 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚935238416896000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚819642434390304 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚935238392955904 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 254697364 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12027196563456 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 210438856 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚847724785551360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 126091613 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚831120707973634 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚859246115302912 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11479181558542 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚657 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 465162592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚831121632988206 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11478273583330 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚859247043290624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 210438856 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 210438858 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 254697364 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚847724767548416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚819642451429664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11521347754496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚859247043290624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61014268 _runtime‚11940.641327366 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚847724767548416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚655 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 210438858 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 187105881 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784º‹ˆí4Þ)/ _wandbruntime‚11940  _runtime‚11940Âk½›ß)Ú 2loss/SRqqt4/assignment_loss‚1.1684191516290108 /loss/FBt1/detection_loss‚0.6944512190510346 /loss/FBt3/detection_loss‚4.044845968484878cÄ-ìÅ5 epoch‚8 /loss/FRt2/assignment_loss‚2.836648328182025 /loss/FRt2/detection_loss‚1.9862319011336718 .loss/FRt3/detection_loss‚1.329534674548123 0loss/SRqqt3/detection_loss‚0.944818789973855 _runtime‚11940.643054444 .loss/FBt2/detection_loss‚2.808274030685425 .loss/FBt4/detection_loss‚2.392706726988157 0loss/FRt4/assignment_loss‚2.6653536508480706 2loss/SRqqt2/assignment_loss‚1.6264263403912385 2loss/FBt2/assignment_loss‚0.060508366123919834 2loss/FBt3/assignment_loss‚0.002042389241978526 0loss/FRt1/assignment_loss‚2.4778686457210117 1loss/FBt1/assignment_loss‚0.05567099558667377 1loss/FBt4/assignment_loss‚0.34405000932876345 /loss/FRt1/detection_loss‚1.6226264126650576 %loss/total_loss‚40.94192631602121 % _timestamp‚1.7885078828660681e+09 /loss/FRt4/detection_loss‚2.9818299089868865 1loss/SRqqt1/detection_loss‚1.7498636543750763 0loss/SRqqt2/detection_loss‚2.663740354279677 1loss/SRqqt4/detection_loss‚2.4063693324724835 _step‚827 /loss/FRt3/assignment_loss‚1.384846321524006 2loss/SRqqt1/assignment_loss‚1.7760817424353699 2loss/SRqqt3/assignment_loss‚0.9187174013648182 trainer/global_step‚10049²h¦eà)ß 2loss/FBt3/assignment_loss‚0.002042389241978526 0loss/FRt1/assignment_loss‚2.4778686457210117 1loss/FBt1/assignment_loss‚0.05567099558667377 1loss/FBt4/assignment_loss‚0.34405000932876345 /loss/FRt1/detection_loss‚1.6226264126650576 %loss/total_loss‚40.94192631602121 % _timestamp‚1.7885078828660681e+09 /loss/FRt4/detection_loss‚2.9818299089868865 1loss/SRqqt1/detection_loss‚1.7498636543750763 0loss/SRqqt2/detection_loss‚2.663740354279677 1loss/SRqqt4/detection_loss‚2.4063693324724835 _step‚827 /loss/FRt3/assignment_loss‚1.384846321524006 2loss/SRqqt1/assignment_loss‚1.7760817424353699 2loss/SRqqt3/assignment_loss‚0.9187174013648182 trainer/global_step‚10049 2loss/SRqqt4/assignment_loss‚1.1684191516290108 /loss/FBt1/detection_loss‚0.6944512190510346 /loss/FBt3/detection_loss‚4.0448459684848785 epoch‚8 /loss/FRt2/assignment_loss‚2.836648328182025 /loss/FRt2/detection_loss‚1.9862319011336718 .loss/FRt3/detection_loss‚1.329534674548123 0loss/SRqqt3/detection_loss‚0.944818789973855 _runtime‚11940.643054444 .loss/FBt2/detection_loss‚2.808274030685425 .loss/FBt4/detection_loss‚2.392706726988157 0loss/FRt4/assignment_loss‚2.6653536508480706 2loss/SRqqt2/assignment_loss‚1.6264263403912385 2loss/FBt2/assignment_loss‚0.060508366123919834»¸™ g’á)‚(:‡ ùíéÔåš°m gpu.0.gpu‚28.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚153.638' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.4095 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èH}mâ)‚(:â ùíéÔà˜Ön) disk./.usagePercent‚75.76180631829688 disk.sdb.out‚1333.259765625 disk.sda2.in‚8.515625 disk.md0.out‚ 1282.45703125 network.sent‚ 2.6722214e+07 disk.md0p1.in‚ 12.296875 disk.sda2.out‚1333.259765625 disk.rbd2.out‚ 3.75341796875 disk.rbd0.out‚ 3.25341796875! disk.dm-0.out‚170332.07421875 disk.sdb.in‚3.78125 disk.nvme0n1.in‚ 939.6640625 disk.nvme1n1.in‚ 744.57421875 disk.rbd2.in‚0 proc.memory.rssMB‚3013.625 network.recv‚ 3.884154e+06 disk.md0p1.out‚ 1282.45703125" disk.nvme0n1.out‚ 116749.734375# disk.nvme1n1.out‚53582.33984375 disk.sdb2.in‚3.78125 disk.sda.out‚1333.259765625$ memory_percent‚92.03391075134277( proc.memory.availableMB‚ 1305.1640625* proc.memory.percent‚18.393707275390625 proc.cpu.threads‚17 disk.sda.in‚8.515625 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.dm-0.in‚ 1684.23828125% disk./.usageGB‚2709.6091651916504 disk.sdb2.out‚1333.259765625 disk.sda1.out‚0 disk.sdb1.out‚0 disk.md0.in‚ 12.296875 cpu‚25.206292384174567“Égã)‚(:Ü ˆîéÔ£¦Ål disk.md0p1.out‚ 1284.359375" disk.nvme0n1.out‚ 116810.421875) proc.memory.availableMB‚ 1305.26171875 disk.sda2.out‚1335.220703125 disk.md0.out‚ 1284.359375 disk.sdb2.in‚3.78125 disk.sdb.out‚1335.220703125 disk.rbd2.out‚ 3.75341796875* proc.memory.percent‚18.393707275390625 cpu‚25.206064635407294 disk.sdb.in‚3.78125 disk.nvme1n1.in‚ 744.57421875# disk.nvme1n1.out‚53661.23828125% disk./.usageGB‚2709.6339111328125 disk.sda1.in‚0 disk.sda1.out‚0! disk.dm-0.out‚170471.66015625 disk.sdb2.out‚1335.220703125 disk.sda.in‚8.515625 disk.sda.out‚1335.220703125 disk.rbd2.in‚0 proc.memory.rssMB‚3013.625 proc.cpu.threads‚17 network.recv‚ 3.889751e+06 disk.rbd0.in‚0) disk./.usagePercent‚75.76249822516876 disk.sda2.in‚8.515625 disk.md0.in‚ 12.296875$ memory_percent‚92.03331470489502 disk.nvme0n1.in‚ 939.6640625 disk.rbd0.out‚ 3.31640625 disk.sdb1.out‚0 disk.md0p1.in‚ 12.296875 disk.dm-0.in‚ 1684.23828125 network.sent‚ 2.6808926e+07 disk.sdb1.in‚0›+3“ä)‚(:ˆ ˆîéÔìÂñl gpu.0.gpu‚32.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚174.903' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.72575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B§Ï<“å)‚(:ˆ —îéÔÁÃúl gpu.0.gpu‚28.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«ÁUnæ)‚(:ã —îéÔÝ“€n disk.md0p1.in‚ 12.296875 proc.cpu.threads‚17$ disk.nvme0n1.out‚116856.64453125 disk.sdb2.in‚3.78125" disk.sdb2.out‚1336.81298828125 network.sent‚ 2.681477e+07! disk.sda.out‚1336.81298828125 disk.rbd0.out‚ 3.31640625 disk.md0p1.out‚ 1285.89453125! disk.sdb.out‚1336.81298828125 disk.nvme1n1.in‚ 744.57421875) disk./.usagePercent‚75.76604703439351 disk.sda1.out‚0' proc.memory.availableMB‚ 1305.015625 proc.memory.rssMB‚3013.625 disk.sda1.in‚0 disk.nvme0n1.in‚ 939.6640625 disk.md0.out‚ 1285.89453125 disk.rbd2.in‚0 cpu‚25.205842484702078 network.recv‚ 3.893688e+06 disk.sda2.in‚8.515625$ memory_percent‚92.03481674194336* proc.memory.percent‚18.393707275390625% disk./.usageGB‚2709.7608337402344 disk.sda.in‚8.515625 disk.rbd0.in‚0 disk.dm-0.in‚ 1684.23828125! disk.dm-0.out‚170522.82421875 disk.sdb.in‚3.78125" disk.sda2.out‚1336.81298828125" disk.nvme1n1.out‚ 53666.1796875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.in‚ 12.296875 disk.rbd2.out‚ 3.81640625¶õC¦4ç)/ _wandbruntime‚11996  _runtime‚11996B‰HÉè)à & lr-AdamW/pg1‚8.456953447576544e-05 & lr-AdamW/pg2‚8.456953447576544e-05 trainer/global_step‚10099 % _timestamp‚1.7885079382278533e+09 _runtime‚11996.005237803 _step‚828y~o†Îé)È & lr-AdamW/pg1‚8.456953447576544e-05 & lr-AdamW/pg2‚8.456953447576544e-05 trainer/global_step‚10099 % _timestamp‚1.7885079382278533e+09 _runtime‚11996.005237803 _step‚828¼6G&4ê)/ _wandbruntime‚11996  _runtime‚11996<ÛÉ_(ë)‚P R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚863300416473088 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 255917435 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12084627132416 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚939659452473856 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11576313719296 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚851724102753792 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12084858011648 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚951744079606272 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 trainer/global_step‚10099 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚11996.006342068 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 255943779 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚823514313562320 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚835048255102484 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚639 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚823514296522960 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚851724120756736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 467370477 % _timestamp‚1.7885079382287855e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 211426696 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚851724102753792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 126682796#DÌí (ì)‡P H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚835048255102484 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚639 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚823514296522960 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚851724120756736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 467370477 % _timestamp‚1.7885079382287855e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 211426696 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚851724102753792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 126682796 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚863300416473088 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 255917435 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12084627132416 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚939659452473856 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11576313719296 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚851724102753792 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12084858011648 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚951744079606272 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 trainer/global_step‚10099 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚11996.006342068 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 255943779 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚823514313562320 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0½$Ðæ4í)/ _wandbruntime‚11997  _runtime‚11997¯z•Y$î)ŒN E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11578345164288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61305181 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚951835505224704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11577435179520 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 187999418 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚939749702118400 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚851805739842560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚951835743253504 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12085827046400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 126694863 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 467415494 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885079393766606e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 211446856 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚863384085006848 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚863383157019136 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 467415494 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 211446856 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚939749678178304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YFDeviceStatsMonitor.on_train_batch_end/reA˜ôîquested_bytes.small_pool.freed‚11534150838438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _runtime‚11997.154547822 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11535058813650 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 188000044 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚851805721839616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 126694861 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚10099 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _step‚830 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚626 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚851805721839616 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61304557 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238028800 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 467389122 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570B¡Â¾'ï)‘N OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 211446858 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚823593314117504 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 255942266 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12086041135104 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 255942266 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _runtime‚11997.154547822 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11535058813650 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 188000044 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚851805721839616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 126694861 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚10099 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _step‚830 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚626 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚851805721839616 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61304557 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238028800 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 467389122 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11578345164288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61305181 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚951835505224704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11577435179520 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 187999418 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚939749702118400 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚851805739842560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚951835743253504 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12085827046400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 126694863 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 467415494 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885079393766606e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 211446856 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚863384085006848 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚863383157019136 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 467415494 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 211446856 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚939749678178304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11534150838438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚867437458189312 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259013632 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚956313976273408 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12143238577152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12143473650688 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 212434698 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11589818795272 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚867438386163712 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11632401144320 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚827465176250160 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 257188681 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11633311115776 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _step‚833 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11632401144320 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61588666 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 469597033 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚944170761636352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚839055012084792 % _timestamp‚1.7885079970729737e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 212434698 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚855805075047936 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚956314235287040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235073536}Í› (€*‡P _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11589818795272 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚867438386163712 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11632401144320 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚827465176250160 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 257188681 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11633311115776 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _step‚833 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11632401144320 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61588666 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 469597033 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚944170761636352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚839055012084792 % _timestamp‚1.7885079970729737e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 212434698 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚855805075047936 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚956314235287040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235073536 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚867438386163712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚944170737696256 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚867437458189312 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259013632 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚956313976273408 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12143238577152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12143473650688 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 212434698 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784Áà«Ë4*/ _wandbruntime‚12056  _runtime‚12056§ª['‚*ŒN WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚839135146952822 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚944260963400704 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 188892478 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚855886676130816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 257187168 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 469668396 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 212454856 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚827544210884064 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚867521126723072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 257213538 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11633522604544 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61595001 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚956405363934208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11590028093546 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 _step‚834 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚636 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚956405620837376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚867521126723072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 188893114 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12144400533504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11634432589312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 469642024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 212454856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11634432589312 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚634 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12144633496576 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 127298111 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61594367 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚944260987340800 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚855886694133760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016u/×&&ƒ*‘N OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 212454858 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 257213538 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11633522604544 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61595001 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚956405363934208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11590028093546 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 _step‚834 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚636 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚956405620837376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚867521126723072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 188893114 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12144400533504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11634432589312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 469642024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 212454856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11634432589312 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚634 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12144633496576 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 127298111 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61594367 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚944260987340800 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚855886694133760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚839135146952822 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚944260963400704 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 188892478 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚855886676130816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 257187168 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 469668396 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 212454856 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚827544210884064 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚867521126723072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 258407239 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚611 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 471876281 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚871575427879936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235944960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 127889296 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚871574499905536 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚859886011336192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚10199 _step‚837 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚859886011336192 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 127889298 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12201796329472 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚859886029339136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚960884055193088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚948682046858752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚12114.641014603 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 258433583 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12202008334336 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚843061769067100 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11645696050380 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185Wl (”*‡P H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚871575427879936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235944960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 127889296 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚871574499905536 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚859886011336192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚10199 _step‚837 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚859886011336192 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 127889298 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12201796329472 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚859886029339136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚960884055193088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚948682046858752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚12114.641014603 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 258433583 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12202008334336 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚843061769067100 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11645696050380 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 258407239 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚611 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 471876281 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 H@DeviceS­´×ÞštatsMonitor.on_train_batch_end/inactive_split.all.current‚639 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 258458440 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 127901363 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 213462858 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚859967630422016 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 471921298 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11690520014336 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚859967648424960 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 189786217 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12203189827584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚871658168439296 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12202965253120 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 213462856 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _runtime‚12115.83490329 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 189786856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚843141903935130 _step‚838 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚948772272563200 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0Rï7`'—*N E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚960975213876224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61885493 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 258458440 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 127901363 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 213462858 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚859967630422016 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 471921298 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11690520014336 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚859967648424960 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 189786217 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12203189827584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚871658168439296 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12202965253120 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 213462856 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _runtime‚12115.83490329 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 189786856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248514560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚843141903935130 _step‚838 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚948772272563200 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 471894926 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 Pm *‚(:â ¼ïéÔÔœ©m* proc.memory.percent‚18.393707275390625 proc.cpu.threads‚17 disk.sdb2.in‚3.78125 disk.md0.out‚ 1303.640625 cpu‚25.203425886129217" disk.sda2.out‚1355.24462890625 disk.nvme0n1.in‚ 939.6640625 disk.md0.in‚ 12.296875 disk.rbd2.in‚0# disk.nvme0n1.out‚117250.1171875 disk.nvme1n1.in‚ 744.57421875# disk.nvme1n1.out‚55175.23046875 network.sent‚ 2.7184937e+07 network.recv‚ 3.944902e+06 disk.sda.in‚8.515625 disk.md0p1.in‚ 12.296875! disk.dm-0.out‚172425.34765625% disk./.usageGB‚2710.6113624572754! disk.sda.out‚1355.24462890625 disk.sda1.out‚0 disk.rbd0.out‚ 3.37939453125 disk.sdb.in‚3.78125 disk.sda2.in‚8.515625 disk.rbd2.out‚ 3.81640625$ memory_percent‚92.04044342041016& proc.memory.availableMB‚ 1304.09375" disk.sdb2.out‚1355.24462890625 disk.rbd0.in‚0! disk.sdb.out‚1355.24462890625) disk./.usagePercent‚75.78982817329589 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0p1.out‚ 1303.640625 disk.sda1.in‚0 disk.dm-0.in‚ 1684.23828125 proc.memory.rssMB‚3013.625ËñÖ‘¡*‚(:† ËïéÔäãÉk gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚166.988' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.747 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–C›®k¢*‚(:à ËïéÔõ­–m( proc.memory.availableMB‚ 1304.5859375) disk./.usagePercent‚75.79041203304872 disk.md0.out‚ 1305.015625 disk.sda1.in‚0 disk.dm-0.out‚ 172712.53125 network.sent‚ 2.7190886e+07 disk.sdb.in‚3.78125 disk.sda2.in‚8.515625# disk.nvme0n1.out‚117250.1171875 disk.nvme1n1.in‚ 744.57421875 proc.memory.rssMB‚3013.625 network.recv‚ 3.949035e+06% disk./.usageGB‚2710.6322441101074 disk.md0.in‚ 12.296875" disk.sdb2.out‚1356.68310546875! disk.sda.out‚1356.68310546875 disk.md0p1.in‚ 12.296875 disk.md0p1.out‚ 1305.015625 disk.sdb1.out‚0* proc.memory.percent‚18.393707275390625 proc.cpu.threads‚17 disk.sda.in‚8.515625" disk.nvme1n1.out‚ 55462.4140625 disk.rbd2.out‚ 3.81640625 disk.sdb2.in‚3.78125 disk.sda1.out‚0" disk.sda2.out‚1356.68310546875 disk.rbd0.out‚ 3.37939453125 disk.sdb1.in‚0 disk.dm-0.in‚ 1684.23828125$ memory_percent‚92.03743934631348 cpu‚25.203216795441325! disk.sdb.out‚1356.68310546875 disk.nvme0n1.in‚ 939.6640625 disk.rbd2.in‚0 disk.rbd0.in‚02™ <4£*/ _wandbruntime‚12172  _runtime‚12172xhöȤ* & lr-AdamW/pg1‚7.259756456706033e-05 & lr-AdamW/pg2‚7.259756456706033e-05 trainer/global_step‚10249 $ _timestamp‚1.788508114488709e+09 _runtime‚12172.265988363 _step‚840»+æXÍ¥*Ç trainer/global_step‚10249 $ _timestamp‚1.788508114488709e+09 _runtime‚12172.265988363 _step‚840 & lr-AdamW/pg1‚7.259756456706033e-05 & lr-AdamW/pg2‚7.259756456706033e-05Èo¨+h4¦*/ _wandbruntime‚12172  _runtime‚12172ìŒ<çö§*‚P XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚835366952743920 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚875712469596160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 _runtime‚12172.267204619 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚965453646571008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚863966983630336 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 % _timestamp‚1.7885081144898448e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62168820 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 214450698 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11701573305488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚953193332081152 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 259678485 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 259652141 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 214450698 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 214450696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 474102837 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 128492548 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚875711541621760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚10249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62168820 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 214450698 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11701573305488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚953193332081152 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 259678485 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 259652141 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 214450698 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 214450696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 474102837 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 128492548 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚875711541621760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚10249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231É€HŸ|4©*/ _wandbruntime‚12173  _runtime‚12173æ©6«'ª*ŒN UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12261502084096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 474147828 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚864048602716160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚626 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚864048584713216 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚953283533845504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚953283557785600 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚835445953299104 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11701782603762 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11746607439360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚875795210155520 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚847148660917438 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11702690578974 % _timestamp‚1.7885081154292936e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚875794282167808 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 190679817 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12261732950016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 259676972 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚835445970338464 trainer/global_step‚10249 _runtime‚12173.207122435 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11745697454592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 474174200 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 259703342 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 474174200 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 128504613 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 259703342 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62175204 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 190679191 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚965545290735616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784ñÌZ x«*‘N _runtime‚12173.207122435 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11745697454592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 474174200 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 259703342 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 474174200 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 128504613 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 259703342 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62175204 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 190679191 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚965545290735616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12261502084096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 474147828 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚864048602716160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚626 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚864048584713216 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚953283533845504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚953283557785600 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚835445953299104 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11701782603762 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11746607439360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚875795210155520 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitorEš0Ÿ.on_train_batch_end/requested_bytes.all.allocated‚847148660917438 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11702690578974 % _timestamp‚1.7885081154292936e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚875794282167808 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 190679817 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12261732950016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 259676972 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚835445970338464 trainer/global_step‚10249ÊÝy¸(4¬*/ _wandbruntime‚12173  _runtime‚12173n-¨]­*× /loss/FRt1/assignment_loss‚1.947125067313512 .loss/FRt4/detection_loss‚3.095627863407135 epoch‚8 0loss/FRt4/assignment_loss‚2.2601728725433348 /loss/FRt2/detection_loss‚2.3638604102390155 1loss/SRqqt1/detection_loss‚1.5940995911757152 3loss/FBt3/assignment_loss‚0.0075527827721089125 1loss/SRqqt2/detection_loss‚2.5562764270739122 1loss/FBt4/assignment_loss‚0.12665130876064268 /loss/FRt3/detection_loss‚1.8733410373622297 .loss/FBt4/detection_loss‚2.144508531689644 _step‚843 2loss/SRqqt2/assignment_loss‚1.5184147351167419 1loss/FBt1/assignment_loss‚0.07876282919050036 /loss/FBt1/detection_loss‚0.5911065396035045 2loss/SRqqt1/assignment_loss‚1.2811571057095672 2loss/SRqqt3/assignment_loss‚0.7731915014987213 0loss/SRqqt4/assignment_loss‚1.41273390303456 .loss/FRt1/detection_loss‚1.496141752924554 trainer/global_step‚10249 % _timestamp‚1.7885081154318798e+09 0loss/FRt2/assignment_loss‚2.9762018535818373 1loss/FBt2/assignment_loss‚0.18402041706489258 1loss/SRqqt3/detection_loss‚0.9503078729612753 1loss/SRqqt4/detection_loss‚2.2931643628305003 /loss/FRt3/assignment_loss‚1.483012793110866 .loss/FBt2/detection_loss‚2.384682678259336 .loss/FBt3/detection_loss‚5.479120641946793 $loss/total_loss‚40.8712348791709 _runtime‚12173.208855293©¶œ`b®*Ü 3loss/FBt3/assignment_loss‚0.0075527827721089125 1loss/SRqqt2/detection_loss‚2.5562764270739122 1loss/FBt4/assignment_loss‚0.12665130876064268 /loss/FRt3/detection_loss‚1.8733410373622297 .loss/FBt4/detection_loss‚2.144508531689644 _step‚843 2loss/SRqqt2/assignment_loss‚1.5184147351167419 1loss/FBt1/assignment_loss‚0.07876282919050036 /loss/FBt1/detection_loss‚0.5911065396035045 2loss/SRqqt1/assignment_loss‚1.2811571057095672 2loss/SRqqt3/assignment_loss‚0.7731915014987213 0loss/SRqqt4/assignment_loss‚1.41273390303456 .loss/FRt1/detection_loss‚1.496141752924554 trainer/global_step‚10249 % _timestamp‚1.7885081154318798e+09 0loss/FRt2/assignment_loss‚2.9762018535818373 1loss/FBt2/assignment_loss‚0.18402041706489258 1loss/SRqqt3/detection_loss‚0.9503078729612753 1loss/SRqqt4/detection_loss‚2.2931643628305003 /loss/FRt3/assignment_loss‚1.483012793110866 .loss/FBt2/detection_loss‚2.384682678259336 .loss/FBt3/detection_loss‚5.479120641946793 $loss/total_loss‚40.8712348791709 _runtime‚12173.208855293 /loss/FRt1/assignment_loss‚1.947125067313512 .loss/FRt4/detection_loss‚3.095627863407135 epoch‚8 0loss/FRt4/assignment_loss‚2.2601728725433348 /loss/FRt2/detection_loss‚2.3638604102390155 1loss/SRqqt1/detection_loss‚1.5940995911757152ËΨO’¯*‚(:‡ ÚïéÔ¹l gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚174.562' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.6405 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹³tgh°*‚(:Ý ÚïéÔÙí–m disk.rbd0.in‚0 disk.rbd0.out‚ 3.37939453125 disk.md0p1.out‚ 1306.8515625 disk.md0.in‚ 12.296875 disk.sdb2.in‚3.78125 proc.cpu.threads‚17 disk.sdb.in‚3.78125 disk.sdb1.out‚0 disk.nvme0n1.in‚ 939.6640625 disk.sda.out‚1358.580078125* proc.memory.percent‚18.393707275390625% disk./.usageGB‚2709.6382369995117 disk.md0p1.in‚ 12.296875 disk.sdb.out‚1358.580078125 disk.rbd2.in‚0 disk.sda1.in‚0 disk.dm-0.out‚172871.8671875 disk.dm-0.in‚ 1684.23828125 disk.sda2.out‚1358.580078125) disk./.usagePercent‚75.76261917821218$ disk.nvme0n1.out‚117312.65234375# disk.nvme1n1.out‚55559.21484375( proc.memory.availableMB‚ 1304.4453125 disk.sdb1.in‚0 disk.sda2.in‚8.515625 disk.sda1.out‚0$ memory_percent‚92.03829765319824 cpu‚25.20300802747386 disk.nvme1n1.in‚ 744.57421875 disk.md0.out‚ 1306.8515625 disk.rbd2.out‚ 3.81640625 disk.sdb2.out‚1358.580078125 disk.sda.in‚8.515625 proc.memory.rssMB‚3013.625 network.sent‚ 2.7194044e+07 network.recv‚ 3.952886e+06‹oei±*‚(:Þ éïéÔ£žŒm proc.memory.rssMB‚3013.625! disk.sdb2.out‚1360.5673828125 disk.rbd0.out‚ 3.37939453125! disk.sda2.out‚1360.5673828125 disk.sdb1.out‚0! disk.dm-0.out‚172961.38671875 disk.sda2.in‚8.515625 disk.nvme0n1.in‚ 939.6640625* proc.memory.percent‚18.393707275390625) disk./.usagePercent‚75.76785717028413% disk./.usageGB‚2709.8255729675293 disk.sda.in‚8.515625 disk.sdb.out‚1360.5673828125$ disk.nvme0n1.out‚117401.51171875$ memory_percent‚92.03784465789795 disk.nvme1n1.in‚ 744.57421875 network.sent‚ 2.7301235e+07 disk.rbd2.out‚ 3.81640625 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sdb.in‚3.78125 disk.md0.in‚ 12.296875 cpu‚25.20277964178487 network.recv‚ 3.957999e+06 disk.rbd2.in‚0 disk.sdb2.in‚3.78125 disk.sda1.out‚0 disk.md0p1.in‚ 12.296875 disk.md0.out‚ 1308.7890625) proc.memory.availableMB‚ 1304.51953125 proc.cpu.threads‚17 disk.md0p1.out‚ 1308.7890625 disk.nvme1n1.out‚ 55559.875 disk.sda.out‚1360.5673828125 disk.sdb1.in‚0 disk.dm-0.in‚ 1684.23828125—]`­“²*‚(:ˆ éïéÔ‹¦Œm gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.727' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.18175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö±,‘³*‚(:† øïéÔžËk gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.798' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.4495 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g!f´*‚(:Û øïéÔ¬úâl disk.sdb1.out‚0 disk.md0p1.out‚ 1310.21484375 disk.sda2.in‚8.515625( proc.memory.availableMB‚ 1304.6484375 network.recv‚ 3.962264e+06) disk./.usagePercent‚75.76780714649458 disk.nvme1n1.in‚ 744.57421875 disk.md0p1.in‚ 12.296875! disk.dm-0.out‚173050.66796875$ disk.nvme0n1.out‚117490.54296875 disk.sdb2.out‚1362.060546875 disk.sdb1.in‚0 disk.sdb.out‚1362.060546875 disk.sda2.out‚1362.060546875 disk.nvme0n1.in‚ 939.6640625 proc.memory.rssMB‚3013.625 network.sent‚ 2.7307238e+07 disk.md0.in‚ 12.296875 disk.rbd2.out‚ 3.81640625 disk.sdb2.in‚3.78125 disk.sda1.out‚0 disk.dm-0.in‚ 1684.23828125 disk.sda.in‚8.515625 disk.rbd0.out‚ 3.44189453125$ memory_percent‚92.03705787658691* proc.memory.percent‚18.393707275390625 disk.rbd2.in‚0 cpu‚25.20255286779205 disk.sda1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚2709.8237838745117 disk.nvme1n1.out‚ 55560.125 disk.md0.out‚ 1310.21484375 disk.sda.out‚1362.060546875 disk.rbd0.in‚0 disk.sdb.in‚3.78125rŸî¥4µ*/ _wandbruntime‚12224  _runtime‚12224àòe È¶* & lr-AdamW/pg1‚6.879486901483243e-05 & lr-AdamW/pg2‚6.879486901483243e-05 trainer/global_step‚10299 $ _timestamp‚1.788508167032236e+09 _runtime‚12224.809445034 _step‚844ää¶Í·*Ç $ _timestamp‚1.788508167032236e+09 _runtime‚12224.809445034 _step‚844 & lr-AdamW/pg1‚6.879486901483243e-05 & lr-AdamW/pg2‚6.879486901483243e-05 trainer/global_step‚10299Ìò-‹4¸*/ _wandbruntime‚12224  _runtime‚12224â\(¹*P YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 129095798 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 $ _timestamp‚1.788508167033221e+09 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 260897043 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚868047919918592 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚970023375583744 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11800663419392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚839317832471120 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚970023624111616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 476355739 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚868047919918592 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚879848583337984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12319006808064 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 260923387 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 476382085 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚851074358017352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚879848583337984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 215458698 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚957704593363456 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 215458698 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11801573390848 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11800663419392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _step‚845 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11801573390848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62459050 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚957704617303552 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11756542585592 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 Ý[¤ (º*†P UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 191554848 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62458445 trainer/global_step‚10299 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚868047937921536 _runtime‚12224.810502419 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚879848583337984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 215458698 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚957704593363456 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 215458698 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11801573390848 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11800663419392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _step‚845 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11801573390848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62459050 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚957704617303552 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11756542585592 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 129095798 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 $ _timestamp‚1.788508167033221e+09 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 260897043 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚868047919918592 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚970023375583744 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11800663419392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚839317832471120 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚970023624111616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 476355739 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚868047919918592 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚879848583337984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12319006808064 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 260923387 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 476382085 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚851074358017352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚970114786536448 XJDeviceStat¼?ê]%sMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12319967468544 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 260921874 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚879932251871744 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11801784879616 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11802694864384 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11801784879616 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 260948244 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚617 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚879931323884032 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚970115026662400 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚957794843008000 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 260921874 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 476427102 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 191572096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚851154492885174 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12320183654400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚839396833026304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 191572713 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚615 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _runtime‚12225.749601383 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11802694864384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11757659858870 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚868129557007360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚879931323884032 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚970115026662400 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚957794843008000 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 260921874 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 476427102 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 191572096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚851154492885174 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12320183654400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚839396833026304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 191572713 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚615 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _runtime‚12225.749601383 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11802694864384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11757659858870 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚868129557007360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚970114786536448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12319967468544 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 260921874 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚879932251871744 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11801784879616 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11802694864384 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11801784879616 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 260948244 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚617 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0ÎÆØp4À*/ _wandbruntime‚12225  _runtime‚12225­F§_Á*Ù /loss/FRt3/detection_loss‚1.7157961148441883 1loss/SRqqt2/detection_loss‚2.5078176832199097 0loss/SRqqt4/detection_loss‚2.300253766775131 /loss/FBt1/detection_loss‚0.6907231894521308 /loss/FRt2/detection_loss‚2.2408826623398523 0loss/FRt2/assignment_loss‚3.0515625660236063 1loss/FBt1/assignment_loss‚0.05065815116660052 %loss/total_loss‚38.40211649986596 % _timestamp‚1.7885081679741435e+09 /loss/FRt3/assignment_loss‚1.491367562821037 1loss/FBt2/assignment_loss‚0.08865927411755235 /loss/FBt2/detection_loss‚2.3459737492459163 /loss/FBt3/detection_loss‚2.4019076650341353 trainer/global_step‚10299 1loss/SRqqt1/assignment_loss‚0.916814649747985 1loss/SRqqt2/assignment_loss‚1.522131190598011 2loss/SRqqt3/assignment_loss‚0.8218250892298148 0loss/SRqqt1/detection_loss‚1.981396879880659 1loss/SRqqt3/detection_loss‚1.1540312133234951 0loss/FBt4/assignment_loss‚0.2276120450964072 2loss/FBt3/assignment_loss‚0.006468801481484358 /loss/FRt4/detection_loss‚3.1134105117424675 2loss/SRqqt4/assignment_loss‚1.6896832094838221 /loss/FRt1/detection_loss‚1.7113574508269866 /loss/FBt4/detection_loss‚2.0246785320341587 epoch‚8 0loss/FRt1/assignment_loss‚2.0499890781030423 /loss/FRt4/assignment_loss‚2.297115463277568 _runtime‚12225.751138881 _step‚847f/Þ dÂ*Þ 1loss/FBt1/assignment_loss‚0.05065815116660052 %loss/total_loss‚38.40211649986596 % _timestamp‚1.7885081679741435e+09 /loss/FRt3/assignment_loss‚1.491367562821037 1loss/FBt2/assignment_loss‚0.08865927411755235 /loss/FBt2/detection_loss‚2.3459737492459163 /loss/FBt3/detection_loss‚2.4019076650341353 trainer/global_step‚10299 1loss/SRqqt1/assignment_loss‚0.916814649747985 1loss/SRqqt2/assignment_loss‚1.522131190598011 2loss/SRqqt3/assignment_loss‚0.8218250892298148 0loss/SRqqt1/detection_loss‚1.981396879880659 1loss/SRqqt3/detection_loss‚1.1540312133234951 0loss/FBt4/assignment_loss‚0.2276120450964072 2loss/FBt3/assignment_loss‚0.006468801481484358 /loss/FRt4/detection_loss‚3.1134105117424675 2loss/SRqqt4/assignment_loss‚1.6896832094838221 /loss/FRt1/detection_loss‚1.7113574508269866 /loss/FBt4/detection_loss‚2.0246785320341587 epoch‚8 0loss/FRt1/assignment_loss‚2.0499890781030423 /loss/FRt4/assignment_loss‚2.297115463277568 _runtime‚12225.751138881 _step‚847 /loss/FRt3/detection_loss‚1.7157961148441883 1loss/SRqqt2/detection_loss‚2.5078176832199097 0loss/SRqqt4/detection_loss‚2.300253766775131 /loss/FBt1/detection_loss‚0.6907231894521308 /loss/FRt2/detection_loss‚2.2408826623398523 0loss/FRt2/assignment_loss‚3.0515625660236063ψç€kÃ*‚(:à –ðéÔž¡Õm! disk.sda2.out‚1365.2353515625" disk.nvme1n1.out‚ 55639.9140625 disk.rbd0.in‚0 disk.dm-0.in‚ 1684.23828125 proc.cpu.threads‚17 disk.sda2.in‚8.515625 disk.md0p1.out‚ 1313.2734375$ memory_percent‚92.03917980194092 disk.nvme0n1.in‚ 939.6640625 disk.md0.in‚ 12.296875 disk.rbd0.out‚ 3.4658203125 network.sent‚ 2.7409943e+07$ disk./.usageGB‚2709.607490539551 disk.sda.out‚1365.2353515625 disk.sda1.in‚0 disk.sda1.out‚0! disk.dm-0.out‚173396.73046875) proc.memory.availableMB‚ 1304.30078125 disk.md0p1.in‚ 12.296875$ disk.nvme0n1.out‚117756.81640625 disk.sdb2.in‚3.78125 cpu‚25.20211472655165 disk.rbd2.out‚ 3.81640625 disk.sdb1.in‚0 disk.md0.out‚ 1313.2734375 disk.nvme1n1.in‚ 744.57421875 disk.sda.in‚8.515625 disk.sdb1.out‚0 proc.memory.rssMB‚3013.625* proc.memory.percent‚18.393707275390625) disk./.usagePercent‚75.76175949432327 disk.sdb.out‚1365.2353515625 disk.rbd2.in‚0! disk.sdb2.out‚1365.2353515625 network.recv‚ 3.972447e+06 disk.sdb.in‚3.78125´Àï ‘Ä*‚(:† –ðéÔ¥Íçm gpu.0.gpu‚24.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚123.712' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.928 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ.&?‘Å*‚(:† ¥ðéÔøöÔm gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚155.352' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.838 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ZŽziÆ*‚(:Þ ¥ðéÔ£ûn disk.md0.out‚ 1315.08984375 disk.rbd0.out‚ 3.4658203125 disk.dm-0.in‚ 1684.23828125 disk.sdb.out‚1367.1064453125 disk.sda2.in‚8.515625 disk.nvme0n1.in‚ 939.6640625 disk.nvme1n1.in‚ 744.57421875 disk.sda.in‚8.515625 disk.nvme1n1.out‚ 55718.78125 disk.md0.in‚ 12.296875) proc.memory.availableMB‚ 1304.48046875 cpu‚25.20190656631033! disk.dm-0.out‚173513.08203125 disk.sdb1.in‚0! disk.sdb2.out‚1367.1064453125 network.sent‚ 2.7506542e+07 disk.md0p1.out‚ 1315.08984375! disk.sda2.out‚1367.1064453125 disk.sda.out‚1367.1064453125* proc.memory.percent‚18.393707275390625 disk.sdb2.in‚3.78125 network.recv‚ 3.977157e+06" disk./.usageGB‚2709.5732421875 disk.sda1.out‚0$ disk.nvme0n1.out‚117794.30078125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚ 12.296875 disk.sdb.in‚3.78125) disk./.usagePercent‚75.76080189606587 disk.rbd0.in‚0 disk.rbd2.out‚ 3.81640625$ memory_percent‚92.03808307647705 proc.memory.rssMB‚3013.625 disk.rbd2.in‚0 8 c‘Ç*‚(:† ´ðéԕÇm gpu.0.gpu‚51.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.168' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Em¶mÈ*‚(:â ´ðéÔ§›Þm# disk.nvme1n1.out‚55718.96484375 disk.sda1.out‚0 disk.rbd0.out‚ 3.52880859375) disk./.usagePercent‚75.76426420359908 disk.rbd2.out‚ 3.81640625! disk.sdb2.out‚1368.5712890625 disk.sda.in‚8.515625 disk.rbd2.in‚0 disk.md0p1.in‚ 12.296875 disk.md0p1.out‚ 1316.5078125 disk.nvme1n1.in‚ 744.57421875) proc.memory.availableMB‚ 1304.47265625 cpu‚25.201702609902558 disk.dm-0.out‚ 173513.265625 disk.sda2.in‚8.515625 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1684.23828125$ memory_percent‚92.03813076019287 proc.memory.rssMB‚3013.625* proc.memory.percent‚18.393707275390625 disk.sda.out‚1368.5712890625 network.sent‚ 2.7512543e+07% disk./.usageGB‚2709.6970710754395 network.recv‚ 3.981517e+06 disk.sdb1.in‚0 disk.md0.in‚ 12.296875 disk.sdb2.in‚3.78125 disk.sdb1.out‚0 disk.sdb.in‚3.78125$ disk.nvme0n1.out‚117794.30078125 proc.cpu.threads‚17 disk.sdb.out‚1368.5712890625! disk.sda2.out‚1368.5712890625 disk.nvme0n1.in‚ 939.6640625 disk.md0.out‚ 1316.5078125(Í4É*/ _wandbruntime‚12280  _runtime‚12280 ¨Û?ÉÊ*à & lr-AdamW/pg1‚6.508749118558786e-05 & lr-AdamW/pg2‚6.508749118558786e-05 trainer/global_step‚10349 % _timestamp‚1.7885082228904407e+09 _runtime‚12280.667938308 _step‚848à88[ÎË*È % _timestamp‚1.7885082228904407e+09 _runtime‚12280.667938308 _step‚848 & lr-AdamW/pg1‚6.508749118558786e-05 & lr-AdamW/pg2‚6.508749118558786e-05 trainer/global_step‚10349Ðu-6™4Ì*/ _wandbruntime‚12280  _runtime‚12280YCrÑ2Í*P WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 192447333 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11812419840700 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚872128874209792 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚974593249861120 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚872128892212736 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12377371275264 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚883986553028608 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚883986553028608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62748287 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚843268695158960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 216466696 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12377606348800 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 129699048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259013632 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚621 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 262168289 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11856750844416 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 216466698 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚872128892212736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_traiÜéIÕn_batch_start/oversize_allocations.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11857660815872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 478634987 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62748906 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 trainer/global_step‚10349 $ _timestamp‚1.788508222891519e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚872128874209792 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚855081114999660 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚619 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚883985625054208 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚974593508874752 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11856750844416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 262141945 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚855082040014024 J¾ƒ (Î*†P D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _step‚849 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11857660815872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 262168289 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 192447954 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 478608641 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 129699046 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 216466698 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚872128874209792 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚974593249861120 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚872128892212736 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12377371275264 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚883986553028608 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚883986553028608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62748287 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚843268695158960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 216466696 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12377606348800 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 129699048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259013632 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚621 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 262168289 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11856750844416 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 216466698 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚872128892212736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11857660815872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 478634987 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62748906 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 trainer/global_step‚10349 $ _timestamp‚1.788508222891519e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚872128874209792 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚855081114999660 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚619 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚883985625054208 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚974593508874752 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11856750844416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 262141945 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚855082040014024 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 478680004 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚843347712753504 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885082238615704e+09 _runtime‚12281.639743432 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11858782289408 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 478653632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11813537113978 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11814445089190 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 216486856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚850 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 478680004 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 216486858 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 192465172 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚962306104290304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 129711113 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 129711111 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚884069293587968 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚872210511298560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚872210493295616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11857872304640 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 216486856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 262166776 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚974684650769408 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11858782289408 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚843347729792864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572µÖªÒ¶Ñ*‘N S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚884068365600256 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚607 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚855162174882054 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚872210493295616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11857872304640 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 216486856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 262166776 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚974684650769408 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11858782289408 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚843347729792864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 478680004 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚843347712753504 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885082238615704e+09 _runtime‚12281.639743432 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11858782289408 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 478653632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11813537113978 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11814445089190 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 216486856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚850 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 478680004 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 216486858 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 19<{‚¦a 2465172 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚962306104290304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 129711113 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 129711111 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚884069293587968 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚872210511298560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚888122666770432 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242236416 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚847219574886160 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚888122666770432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 480861543 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 _runtime‚12336.321431689 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 217474696 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚966727163808256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚10399 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63040029 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚876209846503936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 263413191 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11913748240896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚979163284585472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 130302296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63040642 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 217474696 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _step‚853 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚979163042349056 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152Âkœƒ (â*‡P UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚888122666770432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 480861543 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 _runtime‚12336.321431689 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 217474696 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚966727163808256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚10399 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63040029 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚876209846503936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 263413191 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11913748240896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚979163284585472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 130302296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63040642 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 217474696 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _step‚853 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚979163042349056 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11913748240896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11869205070812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 263386847 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚888123594744832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚888122666770432 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242236416 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚847219574886160 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725Õ÷¼J@4ã*/ _wandbruntime‚12337  _runtime‚12337lй’ä*‹N ICDevi¦R'ceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63046385 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12437034527232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 263411678 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 217494858 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 193360123 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 480906534 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚966817413452800 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 $ _timestamp‚1.788508279620458e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 130314363 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚854 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11914869714432 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚966817389512704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12437267490304 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 130314361 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 263438048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚625 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚876291447586816 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚876291447586816 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 480932906 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚876291465589760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚876291465589760 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 217494858 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚847298592480704 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚888205407316480 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 217494856 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚888206335304192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 trainer/global_step‚10399 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11869414369086 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11913959729664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚859168931864362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚859168006849790 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11913959729664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 _runtime‚12337.398371538 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚847298609520064 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11914869714432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 480906534 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 217494856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63045762 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚888206335304192 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 263411678 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 480932906 w=0'å*N UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 263411678 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 217494858 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 193360123 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 480906534 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚966817413452800 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 $ _timestamp‚1.788508279620458e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 130314363 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚854 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11914869714432 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚966817389512704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12437267490304 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 130314361 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 263438048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚625 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚876291447586816 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚876291447586816 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 480932906 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚876291465589760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚876291465589760 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 217494858 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚847298592480704 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚888205407316480 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 217494856 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚888206335304192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 trainer/global_step‚10399 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11869414369086 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11913959729664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚859168931864362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚859168006849790 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11913959729664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 _runtime‚12337.398371538 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚847298609520064 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11914869714432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 480906534 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 217494856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63045762 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚888206335304192 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 263411678 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 480932906 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63046385 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12437034527232Öªm4æ*/ _wandbruntime‚12337  _runtime‚12337hÝD†Yç*Ó /loss/FRt1/assignment_loss‚2.286527980457653 1loss/FBt3/assignment_loss‚0.34823891415726393 .loss/FRt1/detection_loss‚1.904671229644071 0loss/FRt3/assignment_loss‚1.0925051939319557 2loss/SRqqt1/assignment_loss‚1.4073809756338596 /loss/FRt4/detection_loss‚3.0616939457563253 0loss/SRqqt3/detection_loss‚0.911829891115427 _runtime‚12337.404768208 1loss/FBt2/assignment_loss‚0.12498925369260405 0loss/FBt4/assignment_loss‚0.2538731336237561 1loss/SRqqt1/detection_loss‚1.9968492901325225 /loss/FBt1/detection_loss‚0.6243303815922322 /loss/FBt2/detection_loss‚2.8447500988841057 0loss/FBt1/assignment_loss‚0.1477135162378919 /loss/FRt2/detection_loss‚2.3073555327020587 0loss/SRqqt2/detection_loss‚2.887998165119262 .loss/FRt3/detection_loss‚1.641633661490091 %loss/total_loss‚39.62599587621776 trainer/global_step‚10399 % _timestamp‚1.7885082796276975e+09 _step‚855 /loss/FRt4/assignment_loss‚2.265290219050187 2loss/SRqqt2/assignment_loss‚1.7630807510798885 0loss/SRqqt4/assignment_loss‚1.43152968319399 0loss/SRqqt4/detection_loss‚2.492273313658578 epoch‚8 0loss/FRt2/assignment_loss‚2.7027716875076293 1loss/SRqqt3/assignment_loss‚1.082030454860942 /loss/FBt3/detection_loss‚2.2855481803417206 /loss/FBt4/detection_loss‚1.7611304223537445)9à^è*Ø 0loss/FRt3/assignment_loss‚1.0925051939319557 2loss/SRqqt1/assignment_loss‚1.4073809756338596 /loss/FRt4/detection_loss‚3.0616939457563253 0loss/SRqqt3/detection_loss‚0.911829891115427 _runtime‚12337.404768208 1loss/FBt2/assignment_loss‚0.12498925369260405 0loss/FBt4/assignment_loss‚0.2538731336237561 1loss/SRqqt1/detection_loss‚1.9968492901325225 /loss/FBt1/detection_loss‚0.6243303815922322 /loss/FBt2/detection_loss‚2.8447500988841057 0loss/FBt1/assignment_loss‚0.1477135162378919 /loss/FRt2/detection_loss‚2.3073555327020587 0loss/SRqqt2/detection_loss‚2.887998165119262 .loss/FRt3/detection_loss‚1.641633661490091 %loss/total_loss‚39.62599587621776 trainer/global_step‚10399 % _timestamp‚1.7885082796276975e+09 _step‚855 /loss/FRt4/assignment_loss‚2.265290219050187 2loss/SRqqt2/assignment_loss‚1.7630807510798885 0loss/SRqqt4/assignment_loss‚1.43152968319399 0loss/SRqqt4/detection_loss‚2.492273313658578 epoch‚8 0loss/FRt2/assignment_loss‚2.7027716875076293 1loss/SRqqt3/assignment_loss‚1.082030454860942 /loss/FBt3/detection_loss‚2.2855481803417206 /loss/FBt4/detection_loss‚1.7611304223537445 /loss/FRt1/assignment_loss‚2.286527980457653 1loss/FBt3/assignment_loss‚0.34823891415726393 .loss/FRt1/detection_loss‚1.904671229644071×ó ´N“é*‚(:ˆ ÿðéÔ¬“Äk gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚178.629' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.65725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lpâViê*‚(:Þ ÿðéÔŒÐèl disk.dm-0.in‚ 1684.23828125" disk.nvme1n1.out‚ 56583.4453125 disk.rbd2.in‚0) proc.memory.availableMB‚ 1388.62109375) disk./.usagePercent‚75.78989686267857 disk.md0p1.out‚ 1324.30859375 disk.md0.in‚ 12.296875 network.sent‚ 2.7644128e+07 disk.sda1.in‚0 disk.md0.out‚ 1324.30859375# proc.memory.rssMB‚ 2926.66015625 disk.sdb1.in‚0 disk.nvme1n1.in‚ 744.57421875 disk.rbd2.out‚ 3.81640625 network.recv‚ 4.00376e+06% disk./.usageGB‚2710.6138191223145 disk.sdb2.out‚ 1376.69140625 disk.sda.in‚8.515625 disk.md0p1.in‚ 12.296875 disk.sda2.out‚ 1376.69140625 disk.sdb2.in‚3.78125 disk.rbd0.out‚ 3.591796875 disk.sdb1.out‚0! disk.dm-0.out‚174379.16796875 disk.sdb.in‚3.78125 disk.sda2.in‚8.515625$ disk.nvme0n1.out‚117795.72265625 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sdb.out‚ 1376.69140625 disk.nvme0n1.in‚ 939.6640625$ memory_percent‚91.52452945709229* proc.memory.percent‚17.862915992736816 cpu‚25.20042300014407 disk.sda.out‚ 1376.69140625 disk.rbd0.in‚0ˆ°÷‘ë*‚(:† ŽñéÔ´æäk gpu.0.gpu‚5.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚178.914' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.7285 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”Í(ãnì*‚(:ã ŽñéÔÐÜÜl disk.md0p1.in‚ 12.296875 disk.sdb.in‚3.78125 disk.sdb.out‚1378.6357421875 disk.rbd2.in‚0 disk.sda.out‚1378.6357421875 disk.sda1.in‚0 disk.sdb1.in‚0) proc.memory.availableMB‚ 1388.73046875* proc.memory.percent‚17.862915992736816% disk./.usageGB‚2710.5751724243164# memory_percent‚91.5238618850708 cpu‚25.200180656393627 network.recv‚ 4.010132e+06$ disk.nvme0n1.out‚117795.72265625 disk.sdb2.in‚3.78125 disk.nvme0n1.in‚ 939.6640625 disk.md0.out‚ 1326.1796875 disk.sda.in‚8.515625 disk.rbd0.in‚0 network.sent‚ 2.775241e+07 disk.dm-0.in‚ 1684.23828125 disk.nvme1n1.in‚ 744.57421875! disk.nvme1n1.out‚ 56584.078125! disk.sdb2.out‚1378.6357421875 disk.rbd0.out‚ 3.591796875 disk.sdb1.out‚0 disk.md0p1.out‚ 1326.1796875 disk.sda1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚75.78881628482765 disk.sda2.in‚8.515625! disk.sda2.out‚1378.6357421875 disk.md0.in‚ 12.296875# proc.memory.rssMB‚ 2926.66015625! disk.dm-0.out‚174379.80078125 disk.rbd2.out‚ 3.81640625‰õ÷K’í*‚(:‡ ñéÔ¡˜¾m gpu.0.gpu‚9.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.763' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.44075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿Šdî*‚(:Ù ñéÔÃðïm disk.md0.in‚ 12.296875 disk.sda.out‚1380.2744140625 disk.md0p1.out‚1327.75* proc.memory.percent‚17.862915992736816! disk.sda2.out‚1380.2744140625 disk.nvme0n1.in‚ 939.6640625 disk.sdb2.in‚3.78125 network.recv‚ 4.013344e+06 disk.sda2.in‚8.515625! disk.nvme0n1.out‚ 117856.28125 disk.sda1.out‚0 disk.md0p1.in‚ 12.296875% disk./.usageGB‚2710.6339683532715 disk.rbd2.in‚0! disk.sdb2.out‚1380.2744140625) proc.memory.availableMB‚ 1388.77734375# proc.memory.rssMB‚ 2926.66015625 disk.dm-0.in‚ 1684.23828125! disk.dm-0.out‚174841.80078125 disk.rbd2.out‚ 3.81640625 disk.sda.in‚8.515625 disk.sda1.in‚0 disk.sdb1.out‚0 proc.cpu.threads‚17 network.sent‚ 2.7757667e+07 disk.sdb.in‚3.78125 disk.sdb.out‚1380.2744140625# disk.nvme1n1.out‚56985.51953125 disk.rbd0.in‚0 disk.rbd0.out‚ 3.591796875 disk.sdb1.in‚0) disk./.usagePercent‚75.79046024360923 disk.nvme1n1.in‚ 744.57421875 disk.md0.out‚1327.75$ memory_percent‚91.52357578277588 cpu‚25.19997403988979)m†æ‘ï*‚(:† ¬ñéÔ°ù·m gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚144.066' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.0165 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë½ø.sð*‚(:è ¬ñéÔζ—n network.sent‚ 2.776601e+07 disk.rbd2.in‚0 disk.sda1.out‚0 disk.md0p1.in‚ 12.296875! disk.dm-0.out‚174967.69921875$ disk.nvme0n1.out‚117924.68359375 proc.cpu.threads‚17 disk.rbd0.out‚ 3.591796875! disk.sdb.out‚1381.65576171875# proc.memory.rssMB‚ 2926.66015625 disk.sdb2.in‚3.78125" disk.sdb2.out‚1381.65576171875 disk.sdb.in‚3.78125 disk.sda2.in‚8.515625* proc.memory.percent‚17.862915992736816 disk.sda.in‚8.515625 disk.rbd0.in‚0! disk.nvme1n1.out‚ 57043.015625 disk.md0.in‚ 12.296875$ memory_percent‚91.52283668518066 cpu‚25.199710906594735 disk.sdb1.out‚0" disk.sda2.out‚1381.65576171875 disk.nvme0n1.in‚ 939.6640625( proc.memory.availableMB‚ 1388.8984375) disk./.usagePercent‚75.79395305605463 disk.rbd2.out‚ 3.81640625 disk.md0p1.out‚ 1329.07421875 network.recv‚ 4.017778e+06$ disk./.usageGB‚2710.758888244629 disk.md0.out‚ 1329.07421875! disk.sda.out‚1381.65576171875 disk.sda1.in‚0 disk.sdb1.in‚0 disk.dm-0.in‚ 1684.23828125 disk.nvme1n1.in‚ 744.574218758ˆY•4ñ*/ _wandbruntime‚12394  _runtime‚12394‰Â`=Éò*à & lr-AdamW/pg1‚5.796175911889761e-05 & lr-AdamW/pg2‚5.796175911889761e-05 trainer/global_step‚10449 % _timestamp‚1.7885083364812293e+09 _runtime‚12394.258794679 _step‚856&¡Ë+Îó*È & lr-AdamW/pg1‚5.796175911889761e-05 & lr-AdamW/pg2‚5.796175911889761e-05 trainer/global_step‚10449 % _timestamp‚1.7885083364812293e+09 _runtime‚12394.258794679 _step‚856Øe¹~Á4ô*/ _wandbruntime‚12394  _runtime‚12394ÝX"·Š õ*P JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚645 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚983733157497856 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚892259708486656 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 264631749 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚892260636461056 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚971238449030656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 218482696 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚892260636461056 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11969835665920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 218482698 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚971238472970752 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 264658093 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚880290782792192 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 483140791 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11968925694464 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11924174350916 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 $ _timestamp‚1.788508336482308e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 264658093 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 130905546 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63331216 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚863094628964276 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 130905548 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784H¾î (ö*†P D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11968925694464 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11924174350916 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 $ _timestamp‚1.788508336482308e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 264658093 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 130905546 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63331216 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚863094628964276 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 130905548 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚645 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚983733157497856 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚892259708486656 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 264631749 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚892260636461056 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚971238449030656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 218482696 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚892260636461056 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11969835665920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 218482698 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚971238472970752 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 264658093 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚880290782792192 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 483140791 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0Ù™‘¬4÷*/ _wandbruntime‚12395  _runtime‚12395ç"'ø*ŒN GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11970957139456 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11925291624194 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 483185808 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 483185808 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12495631671808 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 218502858 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 218502856 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚863175688846670 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚880372401878016 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚858 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 264656580 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚851249489247264 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11926199599406 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚892343377020416 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 218502858 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 194254618 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚863174763832098 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 218502856 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚892343377020416 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 264656580 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚648 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712yL§ù*‘N TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚851249489247264 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11926199599406 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚892343377020416 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 218502858 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 194254618 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚863174763832098 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 218502856 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚892343377020416 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 264656580 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚648 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_ene (u d/active_bytes.all.current‚ 927987712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11970957139456 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11925291624194 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 483185808 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 483185808 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12495631671808 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 218502858 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 218502856 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚863175688846670 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚880372401878016 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚858 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 264656580 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640Úªô4ú*/ _wandbruntime‚12395  _runtime‚12395ݽ¡_û*Ù 1loss/SRqqt4/detection_loss‚2.2397823333740234 _runtime‚12395.344660352 3loss/FBt3/assignment_loss‚0.0016145527867289882 1loss/SRqqt1/detection_loss‚2.1956183314323425 1loss/SRqqt3/detection_loss‚0.9451031309505136 /loss/FRt1/detection_loss‚1.8811020766992521 1loss/SRqqt4/assignment_loss‚1.905613792665077 1loss/FBt1/assignment_loss‚0.12086513588621879 /loss/FRt1/assignment_loss‚1.985008955001831 2loss/SRqqt2/assignment_loss‚1.6349270281692345 trainer/global_step‚10449 _step‚859 /loss/FBt1/detection_loss‚0.6266407478918753 .loss/FRt2/detection_loss‚2.776032737470373 0loss/FRt2/assignment_loss‚3.4005443146734526 0loss/FRt3/assignment_loss‚1.1689010977745056 /loss/FRt4/assignment_loss‚2.601032025538958 /loss/FRt3/detection_loss‚1.7925505604256282 /loss/FBt2/detection_loss‚2.3942112113748277 /loss/FBt4/detection_loss‚2.4122502273983426 &loss/total_loss‚40.582581041400836 1loss/FBt4/assignment_loss‚0.27466938112380984 0loss/SRqqt2/detection_loss‚2.241091344091627 .loss/FBt3/detection_loss‚2.466618537902832 epoch‚8 0loss/FBt2/assignment_loss‚0.1831131008691987 $ _timestamp‚1.788508337567337e+09 2loss/SRqqt1/assignment_loss‚1.3229455860116734 2loss/SRqqt3/assignment_loss‚0.9478740203564149 .loss/FRt4/detection_loss‚3.064470811532094Ö)Ädü*Þ 2loss/SRqqt2/assignment_loss‚1.6349270281692345 trainer/global_step‚10449 _step‚859 /loss/FBt1/detection_loss‚0.6266407478918753 .loss/FRt2/detection_loss‚2.776032737470373 0loss/FRt2/assignment_loss‚3.4005443146734526 0loss/FRt3/assignment_loss‚1.1689010977745056 /loss/FRt4/assignment_loss‚2.601032025538958 /loss/FRt3/detection_loss‚1.7925505604256282 /loss/FBt2/detection_loss‚2.3942112113748277 /loss/FBt4/detection_loss‚2.4122502273983426 &loss/total_loss‚40.582581041400836 1loss/FBt4/assignment_loss‚0.27466938112380984 0loss/SRqqt2/detection_loss‚2.241091344091627 .loss/FBt3/detection_loss‚2.466618537902832 epoch‚8 0loss/FBt2/assignment_loss‚0.1831131008691987 $ _timestamp‚1.788508337567337e+09 2loss/SRqqt1/assignment_loss‚1.3229455860116734 2loss/SRqqt3/assignment_loss‚0.9478740203564149 .loss/FRt4/detection_loss‚3.064470811532094 1loss/SRqqt4/detection_loss‚2.2397823333740234 _runtime‚12395.344660352 3loss/FBt3/assignment_loss‚0.0016145527867289882 1loss/SRqqt1/detection_loss‚2.1956183314323425 1loss/SRqqt3/detection_loss‚0.9451031309505136 /loss/FRt1/detection_loss‚1.8811020766992521 1loss/SRqqt4/assignment_loss‚1.905613792665077 1loss/FBt1/assignment_loss‚0.12086513588621879 /loss/FRt1/assignment_loss‚1.985008955001831Û”ˆÎ²“ý*‚(:ˆ »ñéÔáÞ½l gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.637' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.15925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"°VÝuþ*‚(:ê »ñéÔ¤½±m) disk./.usagePercent‚75.79744874833439 disk.rbd0.in‚0! disk.sdb.out‚1383.60595703125 disk.rbd2.in‚0* proc.memory.percent‚17.862915992736816% disk./.usageGB‚2710.8839111328125 disk.sda.in‚8.515625 disk.rbd0.out‚ 3.591796875 disk.md0p1.out‚ 1330.98046875 disk.md0.in‚ 12.296875 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sdb1.in‚0 disk.sda2.in‚8.515625 disk.rbd2.out‚ 3.81640625 network.recv‚ 4.02159e+06 disk.sda1.out‚0 disk.sdb1.out‚0$ disk.nvme0n1.out‚118013.58203125 disk.nvme1n1.in‚ 744.57421875$ memory_percent‚91.52634143829346 cpu‚25.19947331011764! disk.sda.out‚1383.60595703125 disk.md0p1.in‚ 12.296875! disk.dm-0.out‚175171.73046875 disk.sdb.in‚3.78125 disk.nvme0n1.in‚ 939.6640625) proc.memory.availableMB‚ 1388.32421875" disk.sdb2.out‚1383.60595703125" disk.sda2.out‚1383.60595703125" disk.nvme1n1.out‚ 57158.1484375 network.sent‚ 2.7769611e+07 disk.md0.out‚ 1330.98046875 disk.sdb2.in‚3.78125 disk.dm-0.in‚ 1684.23828125# proc.memory.rssMB‚ 2926.66015625ÓåP.“ÿ*‚(:ˆ ÊñéԲ£k gpu.0.gpu‚12.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.461' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.11525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°–·•l€+‚(:á ÊñéÔؘÍl! disk.sdb2.out‚1384.8701171875 disk.sdb1.in‚0 proc.cpu.threads‚17 network.sent‚ 2.7877695e+07 disk.md0.in‚ 12.296875 disk.sdb2.in‚3.78125 disk.sdb1.out‚0 disk.md0p1.out‚ 1332.1953125 disk.sda2.in‚8.515625 disk.nvme0n1.in‚ 939.6640625# disk.nvme0n1.out‚118061.5859375$ memory_percent‚91.52781963348389) proc.memory.availableMB‚ 1388.08203125 cpu‚25.199236216480333 disk.rbd0.in‚0 disk.sdb.in‚3.78125 disk.sdb.out‚1384.8701171875* proc.memory.percent‚17.862915992736816) disk./.usagePercent‚75.79744821503172 disk.rbd0.out‚ 3.591796875 disk.dm-0.in‚ 1684.23828125 disk.nvme1n1.in‚ 744.57421875 disk.sda.in‚8.515625 disk.sda1.out‚0 disk.dm-0.out‚175335.9296875! disk.sda2.out‚1384.8701171875$ disk./.usageGB‚2710.883892059326 disk.rbd2.in‚0 disk.sda.out‚1384.8701171875 disk.sda1.in‚0 disk.md0p1.in‚ 12.296875# proc.memory.rssMB‚ 2926.66015625 network.recv‚ 4.028123e+06 disk.nvme1n1.out‚ 57274.34375 disk.md0.out‚ 1332.1953125 disk.rbd2.out‚ 3.81640625«¤A“+‚(:ˆ ÙñéÔÆÉÝk gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚134.575' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.64375 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á1„9p‚+‚(:å ÙñéÔŽ§‡m disk.sda1.out‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 939.6640625 disk.nvme1n1.in‚ 744.57421875) disk./.usagePercent‚75.79744821503172! disk.sdb2.out‚1386.5458984375 disk.sda.out‚1386.5458984375 disk.md0p1.in‚ 12.296875 disk.sdb.out‚1386.5458984375 cpu‚25.199014495682945 network.sent‚ 2.788205e+07 disk.rbd0.in‚0" disk.nvme1n1.out‚ 57330.6328125 disk.sdb2.in‚3.78125 disk.dm-0.in‚ 1684.23828125 disk.sdb.in‚3.78125 disk.md0.out‚ 1333.80078125# proc.memory.rssMB‚ 2926.66015625* proc.memory.percent‚17.862915992736816 disk.sda2.in‚8.515625 disk.rbd2.in‚0$ disk.nvme0n1.out‚118235.60546875 disk.md0.in‚ 12.296875 disk.sda1.in‚0! disk.dm-0.out‚175566.23828125! disk.sda2.out‚1386.5458984375 disk.sda.in‚8.515625 disk.md0p1.out‚ 1333.80078125$ memory_percent‚91.52636528015137( proc.memory.availableMB‚ 1388.3203125 disk.rbd0.out‚ 3.591796875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.out‚ 3.81640625 network.recv‚ 4.030921e+06$ disk./.usageGB‚2710.883892059326¦ZÕ4ƒ+/ _wandbruntime‚12449  _runtime‚12449Ì#âÆ„+À % lr-AdamW/pg2‚5.45448898733345e-05 trainer/global_step‚10499 $ _timestamp‚1.788508391558955e+09 _runtime‚12449.336247968 _step‚860 % lr-AdamW/pg1‚5.45448898733345e-05<¸uË…+Å % lr-AdamW/pg2‚5.45448898733345e-05 trainer/global_step‚10499 $ _timestamp‚1.788508391558955e+09 _runtime‚12449.336247968 _step‚860 % lr-AdamW/pg1‚5.45448898733345e-05ÜZ—}4†+/ _wandbruntime‚12449  _runtime‚12449^=ËO(‡+‚P D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚867102310960948 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12025923090944 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 131508796 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚988302778736640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚855121351379920 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 265876651 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚896397678177280 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚884371755086336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 485393693 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 265902995 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12553044483584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚896397678177280 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚896396750202880 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7885083915598462e+09 _runtime‚12449.337190932 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63621831 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚988303029361664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12553271168512 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚884371755086336 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 219490698 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 265876651 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 219490698 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11980051606024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 trainer/global_step‚10499 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚632 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚975749758193152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚855121334340560 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 265902995 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚867101385946584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784>Èg&…&ˆ+‡P =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚896397678177280 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚896396750202880 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7885083915598462e+09 _runtime‚12449.337190932 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63621831 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚988303029361664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12553271168512 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚884371755086336 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 219490698 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 265876651 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 219490698 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11980051606024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 trainer/global_step‚10499 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚632 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚975749758193152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚855121334340560 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 265902995 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚867101385946584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚867102310960948 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12025923090944 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 131508796 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚988302778736640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚855121351379920 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 265876651 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allÈ°¤/ˆocated‚896397678177280 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚884371755086336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 485393693 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 265902995 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12553044483584Ý›ã:h‰+‚(:Ý èñéÔÂòÕm$ disk./.usageGB‚2709.554039001465 disk.sda.in‚8.515625 disk.sda1.in‚0 disk.sdb1.in‚0$ memory_percent‚91.52626991271973 cpu‚25.198752607780392 disk.sdb.in‚3.78125 disk.sdb.out‚1388.259765625 disk.sda2.out‚1388.259765625 disk.md0p1.out‚ 1335.45703125 disk.md0p1.in‚ 12.296875# proc.memory.rssMB‚ 2926.66015625 disk.nvme0n1.out‚ 118283.625 disk.nvme1n1.in‚ 744.57421875 disk.sdb2.out‚1388.259765625 disk.sda.out‚1388.259765625 disk.rbd0.in‚0 disk.rbd0.out‚ 3.591796875* proc.memory.percent‚17.862915992736816) disk./.usagePercent‚75.76026496694718 disk.rbd2.in‚0( proc.memory.availableMB‚ 1388.3359375 disk.sda2.in‚8.515625 disk.nvme0n1.in‚ 939.6640625 disk.md0.in‚ 12.296875 disk.sdb2.in‚3.78125 disk.sda1.out‚0 disk.dm-0.in‚ 1684.23828125 network.sent‚ 2.790094e+07 disk.md0.out‚ 1335.45703125 disk.rbd2.out‚ 3.81640625 disk.sdb1.out‚0! disk.dm-0.out‚175786.40234375 proc.cpu.threads‚17 network.recv‚ 4.035102e+06# disk.nvme1n1.out‚57502.77734375U¥#“Š+‚(:ˆ èñéÔ˜ëÈm gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.213' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.05325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Bßÿ4‹+/ _wandbruntime‚12450  _runtime‚12450v­³A'Œ+ŒN B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 _runtime‚12450.308437755 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚867181520814406 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 219510856 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚896480418736640 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 265927852 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 485438710 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12026134579712 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 195149071 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63627585 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚896480418736640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 485438710 % _timestamp‚1.7885083925307353e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚988394180500992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 485412338 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12554220543488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚625 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 195148446 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 131520863 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 265901482 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚975839959957504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚855200351935104 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _step‚862 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚988394431112704 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚884453356169216 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 131520861 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 485412338 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63628208 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 219510856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12026134579712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚896480418736640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 485438710 % _timestamp‚1.7885083925307353e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚988394180500992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 485412338 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12554220543488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚625 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 195148446 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 131520863 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 265901482 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚975839959957504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚855200351935104 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _step‚862 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚988394431112704 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚884453356169216 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 131520861 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 485412338 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63628208 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 219510856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12026134579712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 _runtime‚12450.308437755 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚867181520814406 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521ÞJsø«4Ž+/ _wandbruntime‚12450  _runtime‚12450ðª_+Ù % _timestamp‚1.7885083925329268e+09 .loss/FBt2/detection_loss‚1.829794164498647 .loss/FBt4/detection_loss‚2.751646041870117 1loss/SRqqt2/assignment_loss‚1.410365262068808 _runtime‚12450.309922331 /loss/FRt1/assignment_loss‚2.382137900307065 /loss/FRt1/detection_loss‚1.7107666616440318 1loss/SRqqt1/detection_loss‚1.8816276514530181 1loss/SRqqt3/detection_loss‚0.8237078788536271 trainer/global_step‚10499 /loss/FRt2/detection_loss‚2.4325427004504343 0loss/FRt2/assignment_loss‚2.6753049991851627 2loss/SRqqt1/assignment_loss‚1.4842096039652823 2loss/SRqqt3/assignment_loss‚1.3277388926206788 2loss/SRqqt4/assignment_loss‚1.7843106774722828 0loss/SRqqt2/detection_loss‚2.383150801062584 /loss/FBt1/detection_loss‚0.7088714303448796 0loss/FRt3/assignment_loss‚1.6104593242590244 1loss/FBt2/assignment_loss‚0.42849091251152155 1loss/FBt4/assignment_loss‚0.09922947491771386 .loss/FBt3/detection_loss‚4.520586311817169 _step‚863 /loss/FRt4/assignment_loss‚2.363682536851792 /loss/FRt3/detection_loss‚1.8909495011545145 /loss/FRt4/detection_loss‚3.2746137494132634 1loss/SRqqt4/detection_loss‚2.9268172330716076 1loss/FBt1/assignment_loss‚0.10648777431813326 2loss/FBt3/assignment_loss‚0.021389774628914893 %loss/total_loss‚42.82888125874028 epoch‚8á…«yd+Þ .loss/FBt2/detection_loss‚1.829794164498647 .loss/FBt4/detection_loss‚2.751646041870117 1loss/SRqqt2/assignment_loss‚1.410365262068808 _runtime‚12450.309922331 /loss/FRt1/assignment_loss‚2.382137900307065 /loss/FRt1/detection_loss‚1.7107666616440318 1loss/SRqqt1/detection_loss‚1.8816276514530181 1loss/SRqqt3/detection_loss‚0.8237078788536271 trainer/global_step‚10499 /loss/FRt2/detection_loss‚2.4325427004504343 0loss/FRt2/assignment_loss‚2.6753049991851627 2loss/SRqqt1/assignment_loss‚1.4842096039652823 2loss/SRqqt3/assignment_loss‚1.3277388926206788 2loss/SRqqt4/assignment_loss‚1.7843106774722828 0loss/SRqqt2/detection_loss‚2.383150801062584 /loss/FBt1/detection_loss‚0.7088714303448796 0loss/FRt3/assignment_loss‚1.6104593242590244 1loss/FBt2/assignment_loss‚0.42849091251152155 1loss/FBt4/assignment_loss‚0.09922947491771386 .loss/FBt3/detection_loss‚4.520586311817169 _step‚863 /loss/FRt4/assignment_loss‚2.363682536851792 /loss/FRt3/detection_loss‚1.8909495011545145 /loss/FRt4/detection_loss‚3.2746137494132634 1loss/SRqqt4/detection_loss‚2.9268172330716076 1loss/FBt1/assignment_loss‚0.10648777431813326 2loss/FBt3/assignment_loss‚0.021389774628914893 %loss/total_loss‚42.82888125874028 epoch‚8 % _timestamp‚1.7885083925329268e+09ß%T'[‘‘+‚(:† ÷ñéÔÑ¿ñl gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.548' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.637 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NUZq’+‚(:æ ÷ñéÔ–æ”m network.recv‚ 4.040491e+06! disk.sdb.out‚1390.32080078125# memory_percent‚91.5250301361084) disk./.usagePercent‚75.76082781457518 disk.sdb.in‚3.78125 disk.rbd2.out‚ 3.81640625 disk.nvme1n1.in‚ 744.57421875% disk./.usageGB‚2709.5741691589355" disk.nvme1n1.out‚ 57599.5703125 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚ 3.591796875! disk.dm-0.out‚175921.05078125* proc.memory.percent‚17.862915992736816$ disk.nvme0n1.out‚118321.48046875 disk.md0.in‚ 12.296875 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.sdb1.out‚0 proc.cpu.threads‚17 network.sent‚ 2.7997144e+07 disk.sda.in‚8.515625 disk.sda2.in‚8.515625" disk.sda2.out‚1390.32080078125 disk.md0.out‚ 1337.453125" disk.sdb2.out‚1390.32080078125! disk.sda.out‚1390.32080078125 disk.dm-0.in‚ 1684.23828125( proc.memory.availableMB‚ 1388.5390625# proc.memory.rssMB‚ 2926.66015625 disk.md0p1.out‚ 1337.453125 disk.nvme0n1.in‚ 939.6640625 disk.sdb2.in‚3.78125 disk.sdb1.in‚0 disk.md0p1.in‚ 12.296875 cpu‚25.198516003425034àº%+““+‚(:ˆ †òéÔõíŽm gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚122.153' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.53825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í“¥i”+‚(:Þ †òéÔ„öÉm disk.nvme1n1.in‚ 744.57421875 disk.md0.out‚ 1338.84375 disk.sda1.out‚0 cpu‚25.19829613495813$ disk./.usageGB‚2709.638500213623 disk.rbd0.out‚ 3.591796875 disk.md0p1.out‚ 1338.84375 disk.nvme0n1.in‚ 939.6640625 disk.rbd2.out‚ 3.81640625( proc.memory.availableMB‚ 1388.5390625* proc.memory.percent‚17.862915992736816 disk.sda2.in‚8.515625 disk.dm-0.out‚ 175981.90625 network.recv‚ 4.046529e+06 disk.rbd0.in‚0 disk.md0p1.in‚ 12.296875 disk.sda1.in‚0 disk.dm-0.in‚ 1684.23828125 disk.sdb.in‚3.78125" disk.sda2.out‚1391.78271484375 disk.md0.in‚ 12.296875! disk.sda.out‚1391.78271484375 disk.sda.in‚8.515625# memory_percent‚91.5250301361084! disk.sdb.out‚1391.78271484375 network.sent‚ 2.8084086e+07 disk.sdb1.in‚0# proc.memory.rssMB‚ 2926.66015625 proc.cpu.threads‚17 disk.rbd2.in‚0" disk.sdb2.out‚1391.78271484375# disk.nvme0n1.out‚118321.6796875" disk.nvme1n1.out‚ 57660.2265625 disk.sdb2.in‚3.78125) disk./.usagePercent‚75.76262653778889 disk.sdb1.out‚0ϵI&‘•+‚(:† •òéÔãìàm gpu.0.gpu‚58.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚180.196' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.049 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W|œÝk–+‚(:à •òéÔ½³°n disk.sdb2.in‚3.78125 cpu‚25.19809514009954 network.sent‚ 2.8087283e+07 disk.sdb1.in‚0 disk.md0p1.out‚ 1340.26953125 disk.rbd2.out‚ 3.81640625# proc.memory.rssMB‚ 2926.66015625 network.recv‚ 4.049943e+06 disk.rbd0.in‚0 disk.sdb.out‚1393.275390625# disk.nvme0n1.out‚118321.9140625 disk.md0.in‚ 12.296875 disk.rbd0.out‚ 3.591796875! disk.nvme1n1.out‚ 57752.390625 disk.sda.out‚1393.275390625$ memory_percent‚91.52958393096924) proc.memory.availableMB‚ 1387.79296875% disk./.usageGB‚2709.7271118164062 disk.sda1.in‚0 disk.sda.in‚8.515625 disk.sda1.out‚0 disk.sdb1.out‚0 disk.dm-0.out‚176074.3046875 disk.sda2.in‚8.515625 disk.nvme1n1.in‚ 744.57421875 proc.cpu.threads‚17 disk.sda2.out‚1393.275390625 disk.rbd2.in‚0* proc.memory.percent‚17.862915992736816 disk.md0p1.in‚ 12.296875 disk.dm-0.in‚ 1684.23828125 disk.sdb2.out‚1393.275390625) disk./.usagePercent‚75.76510415528955 disk.sdb.in‚3.78125 disk.nvme0n1.in‚ 939.6640625 disk.md0.out‚ 1340.26953125Èáõ54—+/ _wandbruntime‚12502  _runtime‚12502¸%Tɘ+à & lr-AdamW/pg1‚5.122630802504939e-05 & lr-AdamW/pg2‚5.122630802504939e-05 trainer/global_step‚10549 % _timestamp‚1.7885084442979546e+09 _runtime‚12502.075214171 _step‚864ùΙ+È & lr-AdamW/pg1‚5.122630802504939e-05 & lr-AdamW/pg2‚5.122630802504939e-05 trainer/global_step‚10549 % _timestamp‚1.7885084442979546e+09 _runtime‚12502.075214171 _step‚864àHS†4š+/ _wandbruntime‚12502  _runtime‚12502¤ /C¤ ›+‚P SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚900533791919104 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 267121553 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚992872543159296 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚888452709377536 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚888452691374592 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12081100544512 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 220498696 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12081100544512 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12082010515968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚900534719893504 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 267147897 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63912389 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12082010515968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚871108142928892 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 132112046 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚888452709377536 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 220498698 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 267121553 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 196025072 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚980261043415552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚859072231107120 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 487620249 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640¼^ (œ+‡P XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 _step‚865 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 196024435 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚900533791919104 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚900533791919104 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 267121553 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚992872543159296 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚888452709377536 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚888452691374592 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12081100544512 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 220498696 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12081100544512 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12082010515968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚900534719893504 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 267147897 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63912389 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12082010515968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚871108142928892 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 132112046 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚888452709377536 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 220498698 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 267121553 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 196025072 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚980261043415552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚859072231107120 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 487620249 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 487691612 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12612729182720 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233834496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚980351269120000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 196042439 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63918955 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚871189202811286 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚900617460452864 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 132124113 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚980351245179904 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 267146384 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12083131989504 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚888534328463360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚859151231662304 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12612939077120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 267146384 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚629 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚871188277796714 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚900616532465152 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 267172754 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 220518856 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 487691612 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12612729182720 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233834496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚980351269120000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 196042439 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMon¶¦ñitor.on_train_batch_end/inactive_split.small_pool.allocated‚63918955 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚871189202811286 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚900617460452864 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 132124113 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚980351245179904 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 267146384 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12083131989504 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚888534328463360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚859151231662304 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12612939077120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 267146384 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚629 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚871188277796714 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚900616532465152 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 267172754 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 220518856 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚892533645665792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚997442369486336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 132715298 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64203372 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 221506696 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12138097940992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚631 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚984772304697856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚904670833635328 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12138097940992 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 196918668 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 268392799 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12092714091244 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 _runtime‚12559.744957838 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚997442624305664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 132715296 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12670295667712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚875115824925564 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚892533663668736 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 268366455 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12091806116240 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚904670833635328 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12138097940992 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 196918668 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 268392799 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12092714091244 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 _runtime‚12559.744957838 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚997442624305664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 132715296 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12670295667712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚875115824925564 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚892533663668736 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 268366455 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12091806116240 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚892533645665792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚997442369486336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 132715298 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64203372 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 221506696 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12138097940992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚631 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚984772304697856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 \ODeviceStatsMonitor.on_train_batch_start/inactive_s{Û’ plit_bytes.large_pool.current‚23940096 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 489944514 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 268391286 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚892615264751616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12092923389518 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚875195034779022 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 132727361 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚870 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 268391286 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12671234853888 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 _runtime‚12560.950063046 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚892615282754560 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 196936524 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 % _timestamp‚1.7885085031717842e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12093831364730 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64209786 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12138309429760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 489944514 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 221526856 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 trainer/global_step‚10599 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12139219414528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64209163 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚875195959793594 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚984862530402304 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 221526858 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚863102128428864 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚997534020062208 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 221526856 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12138309429760 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 132727363 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚863102111389504 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 221526858 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚904753574181376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚997534020062208 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 221526856 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12138309429760 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 132727363 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚863102111389504 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 221526858 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚904753574181376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 489944514 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 268391286 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚892615264751616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12092923389518 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚875195034779022 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 132727361 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚870 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 268391286 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12671234853888 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 _runtime‚12560.950063046 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚892615282754560 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 196936524 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 % _timestamp‚1.7885085031717842e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12093831364730 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64209786 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12138309429760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 489944514 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 221526856 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 trainer/global_step‚10599 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12139219414528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64209163 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚875195959793594 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚984862530402304 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144æmNµ4´+/ _wandbruntime‚12560  _runtime‚12560è¶ÈO_µ+Ù 2loss/SRqqt3/assignment_loss‚0.9356791015622045 1loss/FBt1/assignment_loss‚0.05872337342770341 1loss/SRqqt2/detection_loss‚2.1622234845161437 1loss/SRqqt3/detection_loss‚0.8789477933598452 /loss/FBt3/detection_loss‚2.6591876298189163 /loss/FRt4/assignment_loss‚2.927184263865153 /loss/FBt1/detection_loss‚0.7032737457193434 _step‚871 2loss/SRqqt2/assignment_loss‚1.9309215438365936 2loss/SRqqt4/assignment_loss‚1.5864974211256664 /loss/FRt3/detection_loss‚1.6191194965008726 /loss/FRt4/detection_loss‚3.3863570321173895 /loss/FBt4/detection_loss‚2.1573653055561914 /loss/FRt1/assignment_loss‚2.411389899808307 .loss/FRt2/assignment_loss‚3.25119270790707 2loss/SRqqt1/assignment_loss‚1.6315327676988782 0loss/FBt2/assignment_loss‚0.3590850964152651 /loss/FRt1/detection_loss‚1.4590427982939007 /loss/FRt2/detection_loss‚2.1332671256227926 _runtime‚12560.951383562 epoch‚8 0loss/FRt3/assignment_loss‚1.2971384601390108 1loss/FBt3/assignment_loss‚0.07676330143294763 1loss/SRqqt1/detection_loss‚2.4842249268577214 /loss/FBt2/detection_loss‚2.4186473935842514 trainer/global_step‚10599 % _timestamp‚1.7885085031742861e+09 1loss/FBt4/assignment_loss‚0.24436123978448954 0loss/SRqqt4/detection_loss‚2.448776907645739 %loss/total_loss‚41.22090281659639-†Ëd¶+Þ /loss/FRt1/assignment_loss‚2.411389899808307 .loss/FRt2/assignment_loss‚3.25119270790707 2loss/SRqqt1/assignment_loss‚1.6315327676988782 0loss/FBt2/assignment_loss‚0.3590850964152651 /loss/FRt1/detection_loss‚1.4590427982939007 /loss/FRt2/detection_loss‚2.1332671256227926 _runtime‚12560.951383562 epoch‚8 0loss/FRt3/assignment_loss‚1.2971384601390108 1loss/FBt3/assignment_loss‚0.07676330143294763 1loss/SRqqt1/detection_loss‚2.4842249268577214 /loss/FBt2/detection_loss‚2.4186473935842514 trainer/global_step‚10599 % _timestamp‚1.7885085031742861e+09 1loss/FBt4/assignment_loss‚0.24436123978448954 0loss/SRqqt4/detection_loss‚2.448776907645739 %loss/total_loss‚41.22090281659639 2loss/SRqqt3/assignment_loss‚0.9356791015622045 1loss/FBt1/assignment_loss‚0.05872337342770341 1loss/SRqqt2/detection_loss‚2.1622234845161437 1loss/SRqqt3/detection_loss‚0.8789477933598452 /loss/FBt3/detection_loss‚2.6591876298189163 /loss/FRt4/assignment_loss‚2.927184263865153 /loss/FBt1/detection_loss‚0.7032737457193434 _step‚871 2loss/SRqqt2/assignment_loss‚1.9309215438365936 2loss/SRqqt4/assignment_loss‚1.5864974211256664 /loss/FRt3/detection_loss‚1.6191194965008726 /loss/FRt4/detection_loss‚3.3863570321173895 /loss/FBt4/detection_loss‚2.1573653055561914çLÒ ‘·+‚(:† àòéÔÄÁ•n gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.372' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.093 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k¥â¬m¸+‚(:â àòéÔËû×n disk.nvme1n1.in‚ 744.57421875 disk.md0.in‚ 13.58203125 disk.md0.out‚ 1348.62890625 network.recv‚ 4.072206e+06 disk.rbd2.in‚0 proc.cpu.threads‚17 network.sent‚ 2.8218811e+07 disk.dm-0.in‚ 1684.23828125 disk.sdb.in‚4.390625$ memory_percent‚91.52534008026123 disk.rbd2.out‚ 3.81640625 disk.sdb1.in‚0) proc.memory.availableMB‚ 1388.48828125# proc.memory.rssMB‚ 2926.66015625# disk./.usageGB‚2709.88614654541 disk.sdb2.out‚ 1401.9453125 disk.sda.out‚ 1401.9453125 disk.sda1.out‚0 disk.md0p1.out‚ 1348.62890625 disk.sda2.out‚ 1401.9453125 disk.rbd0.out‚ 3.591796875 disk.md0p1.in‚ 13.58203125# disk.nvme0n1.out‚118322.2890625( disk./.usagePercent‚75.7695508328737 disk.sdb1.out‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda2.in‚ 9.19140625 disk.nvme0n1.in‚ 939.6640625# disk.nvme1n1.out‚58597.53515625* proc.memory.percent‚17.862915992736816 cpu‚25.19709289685811 disk.sdb2.in‚4.390625 disk.sda.in‚ 9.19140625! disk.dm-0.out‚176919.82421875 disk.sdb.out‚ 1401.9453125¸Š“¹+‚(:ˆ ïòéÔßÊm gpu.0.gpu‚52.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.169' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.29225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–ã9Llº+‚(:á ïòéÔ®™ˆn# proc.memory.rssMB‚ 2926.66015625 disk.md0p1.in‚ 13.58203125 disk.nvme0n1.in‚ 939.6640625 disk.sdb2.out‚ 1403.4140625 disk.md0.out‚ 1350.05078125 cpu‚25.196900704123177 disk.sda1.in‚0 disk.rbd0.out‚ 3.591796875 disk.sdb1.out‚0 disk.md0p1.out‚ 1350.05078125 disk.sdb.out‚ 1403.4140625 disk.sda2.in‚ 9.19140625 disk.md0.in‚ 13.58203125* proc.memory.percent‚17.862915992736816 disk.sdb2.in‚4.390625 disk.sda.in‚ 9.19140625 disk.dm-0.in‚ 1684.23828125( proc.memory.availableMB‚ 1388.4765625 proc.cpu.threads‚17 network.sent‚ 2.8326473e+07% disk./.usageGB‚2710.2613525390625 disk.sda1.out‚0 disk.sdb.in‚4.390625) disk./.usagePercent‚75.78004174949213 disk.rbd2.in‚0 disk.sdb1.in‚0! disk.dm-0.out‚177023.94921875 disk.nvme1n1.in‚ 744.57421875 network.recv‚ 4.077582e+06 disk.sda.out‚ 1403.4140625 disk.nvme0n1.out‚ 118322.625 disk.rbd2.out‚ 3.81640625 disk.rbd0.in‚0 disk.sda2.out‚ 1403.4140625# disk.nvme1n1.out‚58701.32421875$ memory_percent‚91.52541160583496Ê|ˆ‘»+‚(:† þòéÔ©Ï€m gpu.0.gpu‚15.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚121.876' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.469 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆÜ\¼+‚(:ô þòéÔn disk.dm-0.in‚ 1684.23828125" disk.sda2.out‚1405.35693359375 disk.sda1.out‚0 disk.sdb1.in‚0 disk.dm-0.out‚177024.1640625# disk.nvme1n1.out‚58701.51953125 disk.rbd2.in‚0" disk.sdb2.out‚1405.35693359375 cpu‚25.19670597385165 network.sent‚ 2.8335069e+07 disk.md0.out‚ 1351.91015625 disk.rbd2.out‚ 3.81640625! disk.sda.out‚1405.35693359375! disk.sdb.out‚1405.35693359375 disk.nvme0n1.in‚ 939.6640625 disk.rbd0.in‚0 disk.sda2.in‚ 9.19140625 disk.sda.in‚ 9.19140625 disk.sda1.in‚0# proc.memory.rssMB‚ 2926.66015625 network.recv‚ 4.082382e+06) disk./.usagePercent‚75.79758911359467$ memory_percent‚91.52548313140869) proc.memory.availableMB‚ 1388.46484375$ disk./.usageGB‚2710.888931274414 disk.md0p1.out‚ 1351.91015625 disk.sdb.in‚4.390625$ disk.nvme0n1.out‚118322.64453125 disk.nvme1n1.in‚ 744.57421875 disk.sdb1.out‚0* proc.memory.percent‚17.862915992736816 proc.cpu.threads‚17 disk.md0p1.in‚ 13.58203125 disk.md0.in‚ 13.58203125 disk.sdb2.in‚4.390625 disk.rbd0.out‚ 3.591796875{\º›’½+‚(:‡ óéÔÒþm gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.266' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.3165 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0܃jVu¾+‚(:ê óéÔ©ßžn disk.sdb.in‚4.390625 disk.sda2.in‚ 9.19140625) disk./.usagePercent‚75.79751402458005 disk.sda1.in‚0 disk.sdb1.in‚0 disk.md0p1.in‚ 13.58203125 network.recv‚ 4.08591e+06$ disk./.usageGB‚2710.886245727539! disk.sdb.out‚1406.98779296875* proc.memory.percent‚17.862915992736816 cpu‚25.19651177527068 proc.cpu.threads‚17 disk.nvme1n1.in‚ 744.57421875 disk.nvme1n1.out‚ 59210.4375 disk.dm-0.in‚ 1684.23828125" disk.sda2.out‚1406.98779296875 disk.nvme0n1.in‚ 939.6640625 disk.rbd2.out‚ 3.81640625 disk.rbd0.in‚0 disk.rbd0.out‚ 3.591796875 disk.sdb1.out‚0$ memory_percent‚91.52548313140869 disk.rbd2.in‚0" disk.sdb2.out‚1406.98779296875! disk.sda.out‚1406.98779296875 disk.sda1.out‚0" disk.nvme0n1.out‚ 118322.734375) proc.memory.availableMB‚ 1388.46484375 network.sent‚ 2.8340356e+07 disk.sda.in‚ 9.19140625 disk.dm-0.out‚ 177533.171875 disk.md0.in‚ 13.58203125# proc.memory.rssMB‚ 2926.66015625 disk.sdb2.in‚4.390625 disk.md0p1.out‚ 1353.4765625 disk.md0.out‚ 1353.4765625R,¢4¿+/ _wandbruntime‚12618  _runtime‚12618'ÊðsÈÀ+ & lr-AdamW/pg2‚4.488675224317324e-05 trainer/global_step‚10649 $ _timestamp‚1.788508561180594e+09 _runtime‚12618.957784817 _step‚872 & lr-AdamW/pg1‚4.488675224317324e-05ßZÍÁ+Ç & lr-AdamW/pg2‚4.488675224317324e-05 trainer/global_step‚10649 $ _timestamp‚1.788508561180594e+09 _runtime‚12618.957784817 _step‚872 & lr-AdamW/pg1‚4.488675224317324e-05èrå…x4Â+/ _wandbruntime‚12618  _runtime‚12618ÀÇÂ}Ã+„P YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 133318548 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1002012145875968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _step‚873 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 trainer/global_step‚ïŽY%10649 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚989283613860352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚908807875351552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 222514698 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12728784737792 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12148591346352 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12147683371348 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚896614599956992 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 222514698 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 222514696 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚908808803325952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚866973990561520 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 492152399 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1002012398598144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 269637701 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚908807875351552 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12194185366016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 % _timestamp‚1.7885085611813493e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12193275394560 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 269637701 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64493606 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 133318546 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12194185366016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚989283589920256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚879121656893508 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚896614617959936 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚879122581907872 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 269611357 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚642 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0©a$¡(Ä+‰P QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚644 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 197812152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚896614617959936 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚879122581907872 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 269611357 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚642 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 133318548 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1002012145875968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _step‚873 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 trainer/global_step‚10649 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚989283613860352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚908807875351552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 222514698 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12728784737792 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12148591346352 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12147683371348 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚896614599956992 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 222514698 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 222514696 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚908808803325952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚866973990561520 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 492152399 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1002012398598144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 269637701 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚908807875351552 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12194185366016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 % _timestamp‚1.7885085611813493e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12193275394560 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 269637701 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64493606 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 133318546 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12194185366016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚989283589920256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚879121656893508 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 197829965 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚896696219042816 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12729950000640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12149708619838 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12194396854784 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 269636188 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚908890615897600 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 269662558 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚989373839564800 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚896696219042816 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 197830604 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 trainer/global_step‚10649 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚908891543885312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚896696237045760 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1002103789565440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚867053008156064 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 % _timestamp‚1.7885085623000524e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚896696237045760 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 269636188 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 492197416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 222534856 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚908891543885312 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 492197416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚989373815624704üˆ²Žù Ç+“N WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 133330613 _step‚874 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 492171044 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚908891543885312 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 492197416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚989373815624704 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 222534856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1002103536856576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 222534858 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64499991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64499354 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚879202716775902 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12148800644626 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ZGDeviceStatsMonitor.on_tr$Û¾_ ain_batch_end/active_bytes.small_pool.allocated‚12195306839552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚879201791761330 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 197829965 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚896696219042816 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12729950000640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12149708619838 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12194396854784 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 269636188 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚908890615897600 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 269662558 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚989373839564800 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚896696219042816 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 197830604 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 trainer/global_step‚10649 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚908891543885312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚896696237045760 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1002103789565440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚867053008156064 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 % _timestamp‚1.7885085623000524e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚896696237045760 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 269636188 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 492197416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 222534856 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚900695572251136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚12677.784219814 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚993794899082752 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚870924853249360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 133921796 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1006581843879936 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚639 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚883129338890180 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64784541 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12204468601460 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12249362819584 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1006582098699264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12787199616512 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚870924870288720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 223522696 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚993794875142656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 223522696 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12250272791040 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 494378955 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚912944917067776 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 198706976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 133921798 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚912945845042176 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 _step‚877 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 223522698 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚912944917067776 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚900695572251136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚900695554248192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 270882603 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 198706337 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 494405301 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 270882603 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 198706337 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 494405301 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚900695572251136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚12677.784219814 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚993794899082752 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚870924853249360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 133921796 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1006581843879936 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚639 YEDeviceStatsMonitor.oÈC³8²n_train_batch_start/requested_bytes.all.allocated‚883129338890180 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64784541 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12204468601460 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12249362819584 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1006582098699264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12787199616512 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚870924870288720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 223522696 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚993794875142656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 223522696 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12250272791040 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 494378955 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚912944917067776 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 198706976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 133921798 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚912945845042176 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 _step‚877 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 223522698 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚912944917067776 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393í¦ˆª+4Ù+/ _wandbruntime‚12678  _runtime‚12678ŸŽø°'Ú+ŽN MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚883208548743638 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 trainer/global_step‚10699 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 494450318 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 494450318 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12205585874946 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885086211976752e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 223542858 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 223542856 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12204677899734 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 494423946 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _runtime‚12678.976556355 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚913028585601536 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12250484279808 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12251394264576 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 223542858 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚871003870843904 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12788361985536 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚900777173334016 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 494423946 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 270881090 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64790885 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚883209473758210 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 270881090 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚993885100847104 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 198724748 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1006673486772736 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12250484279808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚913027657613824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1006673240355328 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚900777173334016 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 198724126 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 270907460 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚913028585601536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 133933861 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚993885124787200 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12251394264576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12788139508224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 _step‚878 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚900777191336960 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 270907460 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚871003887883264C½ù'Û+“N @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 494450318 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12205585874946 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885086211976752e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 223542858 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 223542856 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12204677899734 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 494423946 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _runtime‚12678.976556355 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚913028585601536 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12250484279808 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12251394264576 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 223542858 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚871003870843904 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12788361985536 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚900777173334016 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 494423946 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 270881090 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64790885 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚883209473758210 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 270881090 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚993885100847104 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 198724748 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1006673486772736 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12250484279808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚913027657613824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1006673240355328 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚900777173334016 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 198724126 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 270907460 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚913028585601536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 133933861 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚993885124787200 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12251394264576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12788139508224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 _step‚878 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚900777191336960 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 270907460 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚871003887883264 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚883208548743638 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 trainer/global_step‚10699 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 494450318 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 224530696 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 272101161 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 134525046 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12306360216064 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚917082886758400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12259437881564 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1011151792554496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚874875750015920 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 496631857 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 496658203 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 224530696 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 199601166 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 trainer/global_step‚10749 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1011152049470976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚904776526542336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12260345856568 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 134525048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚887136095872488 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚887135170858124 _step‚881 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚998306184305152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚904776508539392 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 272127505 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚917081958784000 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚904776526542336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725Qêu‹(ê+‰P QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 % _timestamp‚1.7885086744260283e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 224530698 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚904776508539392 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12305450244608 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1011152049470976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚904776526542336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12260345856568 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 134525048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚887136095872488 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚887135170858124 _step‚881 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚998306184305152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚904776508539392 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 272127505 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚917081958784000 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚904776526542336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚998306160365056 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12305450244608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 199600546 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 224530696 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 272101161 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 134525046 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12306360216064 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚917082886758400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12259437881564 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1011151792554496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚874875750015920 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 496631857 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 496658203 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 224530696 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 199601166 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 trainer/global_step‚10749 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12261463130054 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 496703220 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚616 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚904858127625216 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 199618496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚882 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 496676848 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 224550858 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12847034443776 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 272152362 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12307481689600 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1011243444453376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12846811966464O»òÝí+“N MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10749 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1011243444453376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_enT}f<d/inactive_split_bytes.small_pool.freed‚12846811966464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚887216230740518 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12261463130054 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 496703220 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚616 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚904858127625216 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 199618496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚882 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 496676848 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 224550858 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12847034443776 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 272152362 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12307481689600 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚921219928474624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12361537669632 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 225538698 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1015721574204416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 267402240 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65367229 $ _timestamp‚1.788508728010645e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 498911105 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 200495525 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 225538698 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚921219000500224 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚921219928474624 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚878826612703760 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚891141927840432 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 273372407 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12904372079104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚908857480833536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12315315136672æLþ+ŠP [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1015721574204416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 267402240 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65367229 $ _timestamp‚1.788508728010645e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 498911105 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 200495525 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 225538698 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚921219000500224 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚921219928474624 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚878826612703760 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚891141927840432 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 273372407 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12904372079104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚908857480833536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12315315136672 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1002817469527552 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 273372407 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12362447641088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚660 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12904128616960 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚908857480833536 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚891142852854796 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚921219000500224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚10799 _runtime‚12785.788264012 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚921219928474624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12361537669632 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 225538698õ" “$4ÿ+/ _wandbruntime‚12786  _runtime‚12786´øç'€,N OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚649 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 trainer/global_step‚10799 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 200514066 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚651 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚908939099919360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 135140361 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12317340385162 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 200513415 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12363569114624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1015812998472192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12363569114624 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _runtime‚12786.946640934 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65373054 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 498956122 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 225558856 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚878905647337664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12905537074688 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚921302669033984 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚921301741046272 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12362659129856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 225558858 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚891222062708254 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 273397264 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚908939099919360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚878905630298304 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 498929750 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 273397264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚891222987722826 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 498956122 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1002907671291904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚921302669033984 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 225558858 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570õó‹',”N ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 135140361 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12317340385162 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 200513415 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12363569114624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1015812998472192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12363569114624 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _runtime‚12786.946640934 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65373054 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 498956122 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 225558856 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚878905647337664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12905537074688 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚921302669033984 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚921301741046272 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12362659129856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 225558858 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚891222062708254 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 273397264 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚908939099919360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚878905630298304 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 498929750 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 273397264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚891222987722826 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 498956122 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1002907671291904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚921302669033984 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 225558858 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚649 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 trainer/global_step‚10799 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 200514066 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚651 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚908939099919360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185öûOÔ4‚,/ _wandbruntime‚12786  _runtime‚127868~©ƒ\ƒ,Ö /loss/FBt1/assignment_loss‚0.146371261580224 3loss/FBt3/assignment_loss‚0.0022111409683323777 /loss/FRt3/detection_loss‚1.8350129266057098 1loss/SRqqt3/detection_loss‚0.8984985695068919 1loss/SRqqt4/detection_loss‚2.4545292366634714 2loss/SRqqt4/assignment_loss‚1.5466733290390535 /loss/FRt2/detection_loss‚2.4203820975031705 % _timestamp‚1.7885087291706512e+09 0loss/FRt2/assignment_loss‚3.3530064076185226 1loss/FBt4/assignment_loss‚0.11344834385489674 .loss/FBt3/detection_loss‚2.678883214791616 epoch‚8 trainer/global_step‚10799 2loss/SRqqt1/assignment_loss‚1.1887940744990888 0loss/SRqqt1/detection_loss‚2.335121233826098 $loss/total_loss‚39.5916717152787 _runtime‚12786.94775719 /loss/FRt4/assignment_loss‚2.014048065130527 2loss/SRqqt2/assignment_loss‚1.1453558783076945 2loss/SRqqt3/assignment_loss‚0.9459963991855408 1loss/FBt2/assignment_loss‚0.30475361771833476 .loss/FRt1/detection_loss‚1.799856269324947 1loss/SRqqt2/detection_loss‚2.1932374412013638 /loss/FBt1/detection_loss‚0.7161489225529496 _step‚887 /loss/FRt1/assignment_loss‚2.288237648540073 0loss/FRt3/assignment_loss‚1.3393309335959585 .loss/FRt4/detection_loss‚2.788894772529602 /loss/FBt2/detection_loss‚2.8371306790245905 .loss/FBt4/detection_loss‚2.245749251710044Ö ïma„,Û 3loss/FBt3/assignment_loss‚0.0022111409683323777 /loss/FRt3/detection_loss‚1.8350129266057098 1loss/SRqqt3/detection_loss‚0.8984985695068919 1loss/SRqqt4/detection_loss‚2.4545292366634714 2loss/SRqqt4/assignment_loss‚1.5466733290390535 /loss/FRt2/detection_loss‚2.4203820975031705 % _timestamp‚1.7885087291706512e+09 0loss/FRt2/assignment_loss‚3.3530064076185226 1loss/FBt4/assignment_loss‚0.11344834385489674 .loss/FBt3/detection_loss‚2.678883214791616 epoch‚8 trainer/global_step‚10799 2loss/SRqqt1/assignment_loss‚1.1887940744990888 0loss/SRqqt1/detection_loss‚2.335121233826098 $loss/total_loss‚39.5916717152787 _runtime‚12786.94775719 /loss/FRt4/assignment_loss‚2.014048065130527 2loss/SRqqt2/assignment_loss‚1.1453558783076945 2loss/SRqqt3/assignment_loss‚0.9459963991855408 1loss/FBt2/assignment_loss‚0.30475361771833476 .loss/FRt1/detection_loss‚1.799856269324947 1loss/SRqqt2/detection_loss‚2.1932374412013638 /loss/FBt1/detection_loss‚0.7161489225529496 _step‚887 /loss/FRt1/assignment_loss‚2.288237648540073 0loss/FRt3/assignment_loss‚1.3393309335959585 .loss/FRt4/detection_loss‚2.788894772529602 /loss/FBt2/detection_loss‚2.8371306790245905 .loss/FBt4/detection_loss‚2.245749251710044 /loss/FBt1/assignment_loss‚0.146371261580224÷’õœ“…,‚(:ˆ ÁôéÔ°ù¨m gpu.0.gpu‚53.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚152.567' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.14175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…¢ào†,‚(:ä ÁôéÔæêýn network.sent‚ 2.8726231e+07) disk./.usagePercent‚75.76955989901893 disk.md0.in‚ 13.58203125 disk.dm-0.out‚179622.7265625 disk.sdb.out‚ 1426.5703125 cpu‚25.19413966227792 disk.sda2.in‚ 9.19140625 disk.md0.out‚ 1372.31640625 disk.sdb1.out‚0 disk.md0p1.out‚ 1372.31640625$ memory_percent‚91.52665138244629 proc.cpu.threads‚17 network.recv‚ 4.140158e+06$ disk.nvme0n1.out‚118890.95703125 disk.rbd2.out‚ 3.81640625 disk.sdb2.in‚4.390625 disk.sda1.in‚0 disk.sdb1.in‚0 disk.md0p1.in‚ 13.58203125* proc.memory.percent‚17.862915992736816 disk.rbd0.out‚ 3.591796875 disk.nvme0n1.in‚ 939.6640625 disk.nvme1n1.in‚ 744.57421875 disk.rbd2.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0% disk./.usageGB‚2709.8864707946777 disk.sda2.out‚ 1426.5703125# disk.nvme1n1.out‚60731.76953125 disk.sdb2.out‚ 1426.5703125 disk.sda.out‚ 1426.5703125 disk.dm-0.in‚ 1684.23828125 disk.sdb.in‚4.390625( proc.memory.availableMB‚ 1388.2734375 disk.sda.in‚ 9.19140625# proc.memory.rssMB‚ 2926.66015625H ­‘‡,‚(:† ÐôéÔç³Ëm gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚172.552' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.138 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄiÓ`ˆ,‚(:Õ ÐôéÔÁ¥n disk.dm-0.in‚ 1684.23828125 disk.sda2.in‚ 9.19140625 disk.md0.in‚ 13.58203125 disk.md0p1.in‚ 13.58203125 disk.md0p1.out‚ 1373.62890625 disk.sda2.out‚ 1427.9375 disk.nvme1n1.in‚ 744.57421875$ memory_percent‚91.52989387512207 network.sent‚ 2.8823403e+07 network.recv‚ 4.145502e+06 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-0.out‚ 179716.4375 disk.rbd2.out‚ 3.81640625 disk.sdb2.in‚4.390625( proc.memory.availableMB‚ 1387.7421875 disk.sda1.out‚0 disk.sdb.in‚4.390625 disk.sda.out‚ 1427.9375 disk.nvme0n1.in‚ 939.6640625 proc.cpu.threads‚17) disk./.usagePercent‚75.76955680586349$ disk./.usageGB‚2709.886360168457$ disk.nvme0n1.out‚118890.95703125# proc.memory.rssMB‚ 2926.66015625 cpu‚25.193955396944865 disk.sdb.out‚ 1427.9375 disk.md0.out‚ 1373.62890625 disk.rbd2.in‚0 disk.sdb2.out‚ 1427.9375 disk.sda.in‚ 9.19140625 disk.rbd0.out‚ 3.591796875 disk.sdb1.in‚0 disk.sdb1.out‚0# disk.nvme1n1.out‚60825.48046875* proc.memory.percent‚17.862915992736816»†\s’‰,‚(:‡ ßôéÔþ³´m gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.562' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.6405 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª\[qoŠ,‚(:ä ßôéÔܹžn disk.sdb2.in‚4.390625 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sdb.in‚4.390625 disk.md0p1.out‚ 1375.62890625 disk.md0.in‚ 13.58203125 disk.rbd2.in‚0$ memory_percent‚91.52922630310059) proc.memory.percent‚17.86370277404785 proc.cpu.threads‚17 disk.sda.out‚1430.013671875 disk.sda1.out‚0 disk.md0p1.in‚ 13.58203125 disk.sdb.out‚1430.013671875 disk.sda2.in‚ 9.19140625 disk.sda2.out‚1430.013671875 network.sent‚ 2.8829205e+07 disk.nvme0n1.in‚ 939.6640625$ disk.nvme0n1.out‚118891.20703125 disk.rbd2.out‚ 3.81640625 cpu‚25.1937672950756 disk.rbd0.out‚ 3.591796875# disk.nvme1n1.out‚60883.54296875 disk.sdb2.out‚1430.013671875 disk.dm-0.out‚ 179774.75 disk.md0.out‚ 1375.62890625% disk./.usageGB‚2709.9487075805664 disk.sda.in‚ 9.19140625 disk.sda1.in‚0 disk.dm-0.in‚ 1684.23828125 disk.nvme1n1.in‚ 744.57421875( proc.memory.availableMB‚ 1387.8515625" proc.memory.rssMB‚ 2926.7890625 network.recv‚ 4.149603e+06) disk./.usagePercent‚75.77130006560049Ñd×Ó4‹,/ _wandbruntime‚12838  _runtime‚1283860@PWŒ,Á & lr-AdamW/pg1‚3.341612957326978e-05 & lr-AdamW/pg2‚3.341612957326978e-05 ðd+”ptrainer/global_step‚10849 # _timestamp‚1.78850878049735e+09 _runtime‚12838.274481223 _step‚888' çÌ,Æ & lr-AdamW/pg1‚3.341612957326978e-05 & lr-AdamW/pg2‚3.341612957326978e-05 trainer/global_step‚10849 # _timestamp‚1.78850878049735e+09 _runtime‚12838.274481223 _step‚888øŒUð‡4Ž,/ _wandbruntime‚12838  _runtime‚12838? (,†P TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 % _timestamp‚1.7885087804981592e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 501164007 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65659458 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12962990916608 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚925356042216448 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 135731546 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 274590965 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 201390366 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 135731548 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65658820 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 233847808 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 226546696 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1020291511818752 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 274590965 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12417625094656 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12418535066112 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚638 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚912938435124736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 274617309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12371192391780 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 226546698 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚925356042216448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12372100366784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1007328730809856 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12417625094656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 209907712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 274617309 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚912938435124736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 trainer/global_step‚10849 _step‚889 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚912938417121792 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚912938417121792 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 201391006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 226546696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚925356970190848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚882777492430960 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1020291745666560 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 501137661 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚895149609837104 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚895148684822740 _runtime‚12838.278887173 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚925356970190848 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368ÙAŒ(,‹P \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚882777492430960 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1020291745666560 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 501137661 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚895149609837104 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚895148684822740 _runtime‚12838.278887173 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚925356970190848 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 % _timestamp‚1.7885087804981592e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 501164007 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65659458 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12962990916608 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚925356042216448 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 135731546 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 274590965 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 201390366 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 135731548 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65658820 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 233847808 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 226546696 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1020291511818752 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 274590965 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12417625094656 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12418535066112 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚638 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚912938435124736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 274617309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12371192391780 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 226546698 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚925356042216448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12372100366784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1007328730809856 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12417625094656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 209907712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 274617309 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚912938435124736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 trainer/global_step‚10849 _step‚889 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚912938417121792 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚912938417121792 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 201391006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 226546696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚925356970190848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0ù˜´°†4‘,/ _wandbruntime‚12839  _runtime‚12839U»að'’,N [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚913020054210560 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65664592 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚882856527064864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 274615796 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 274642166 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12373217640270 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚913020036207616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12419656539648 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12419656539648 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 201408203 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 274642166 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1007418956514304 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12964184515584 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚913020036207616 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65665229 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12963976718336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12418746554880 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 501209024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 501209024 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1020383164969984 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚895229744705134 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 226566856 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 501182652 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 207797248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚925439710750208 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12418746554880 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚895228819690562 _step‚890 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 226566856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 274615796 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚925438782762496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚913020054210560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _runtime‚12839.238645287 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 226566858 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231737344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚10849 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885087814606404e+09 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12372309665058 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615:¶Êôñ“,•N QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 274642166 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1007418956514304 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12964184515584 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚913020036207616 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65665229 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12963976718336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VCDeviceStatsMonitor.on_train_batch_eÇêCÄ* nd/active_bytes.small_pool.freed‚12418746554880 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 501209024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 501209024 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1020383164969984 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚895229744705134 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 226566856 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 501182652 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 207797248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚925439710750208 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12418746554880 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚895228819690562 _step‚890 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 226566856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 274615796 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚925438782762496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚913020054210560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _runtime‚12839.238645287 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 226566858 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231737344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚10849 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885087814606404e+09 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12372309665058 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚913020054210560 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65664592 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚882856527064864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 274615796 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 274642166 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12373217640270 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚913020036207616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12419656539648 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12419656539648 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 201408203úÅ…—Ò4”,/ _wandbruntime‚12839  _runtime‚12839DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1185 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 503794670 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚615 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚570 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 227950140 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚918209012311552 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 202692589 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 698106808 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10880 ]JDeviceStatsMonitor.on_validation_batch_staò\pvˆrt/active_bytes.small_pool.freed‚12477762396672 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13684416521ÀÏñ‡v*§,ðT `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚12477762396672 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1187728393 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32006 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚13024684750848 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1195376640 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1185 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 503794670 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚615 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚570 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 227950140 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚918209012311552 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 202692589 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 698106808 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10880 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚12477762396672 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13684416521 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3725 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚12478445435904 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 683039232 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 170501632 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 503773957 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1185 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14401142784 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚918209030314496 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚887776934610384ü-§‚.4¨,/ _wandbruntime‚12933  _runtime‚12933 *2Ô})©,÷R NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚887827377754928 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20711 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚918263273218560 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1187728393 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14401142784 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 275853342 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚930741525466112 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 701042176 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698106808 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1185 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32006 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 275853342 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 275832631 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 409073152 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 503818960 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 701042176 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚930741525466112 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚918263291221504 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13205766144 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 227965616 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚44 ePDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚1013029331041792 _step‚893 \NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚ 136595728 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚12478935286784 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚227 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚930742226508288 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 683039232 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 503798247 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 227965618 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚66111953 ^IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚1026054505643520 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 170501632 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚12432285646372 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚615 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚930742226508288 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 227965618 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚887827360715568 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚66111726 trainer/global_step‚10880 _runtime‚12933.644265267 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14401142784 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 14135738368 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚12478252247552 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚229 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 275853342 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 275832631 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 409073152 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 503818960 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 701042176 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚930741525466112 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚918263291221504 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13205766144 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 227965616 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚44 ePDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚1013029331041792 _step‚893 \NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚ 136595728 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚12478935286784 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚227 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚930742226508288 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 683039232 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 503798247 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 227965618 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚66111953 ^IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚1026054505643520ý¯ì¬„«,‚(:… ×õéÔ×€n gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚183.59' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.8975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œŒw¬,‚(:ë ˜C`zh×õéÔ¼¶Án disk.sdb.out‚1443.099609375 disk.sda2.in‚ 9.19140625 disk.sdb2.out‚1443.099609375) proc.memory.percent‚18.38686466217041 cpu‚25.214871892990587 network.recv‚ 4.184441e+06# disk.nvme1n1.out‚62165.17578125 disk.md0.in‚ 13.58203125 disk.md0.out‚ 1388.2109375 disk.sdb1.out‚0) proc.memory.availableMB‚ 1302.04296875 network.sent‚ 2.8976213e+07 disk.sda2.out‚1443.099609375 disk.nvme1n1.in‚ 744.57421875 disk.md0p1.in‚ 13.58203125! disk.dm-0.out‚181057.43359375$ memory_percent‚92.05296039581299 proc.cpu.threads‚17% disk./.usageGB‚2710.6313133239746# disk.nvme0n1.out‚118892.2578125 disk.rbd0.in‚0 disk.rbd0.out‚ 3.591796875 disk.sdb1.in‚0) disk./.usagePercent‚75.79038600787888 disk.nvme0n1.in‚ 939.6640625 disk.rbd2.out‚ 3.81640625 disk.sdb2.in‚4.390625 disk.sda1.in‚0 disk.md0p1.out‚ 1388.2109375 disk.sdb.in‚4.390625 disk.dm-0.in‚ 1684.23828125 disk.rbd2.in‚0 disk.sda.in‚ 9.19140625 disk.sda.out‚1443.099609375 disk.sda1.out‚0# proc.memory.rssMB‚ 3012.50390625âßÞq4­,/ _wandbruntime‚12949  _runtime‚12949‚ÆO{®,ƒ¦ =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2759627401828766 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4629629850387573 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25430285930633545 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4620808959007263 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.987500011920929 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.46021032333374023 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 )loss/val_total_loss‚39.98426035405653 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.44514259696006775 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1468253880739212 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24350202083587646 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9870368838310242 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4228779077529907 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.37021833658218384 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 particle/accuracy_1_of_2‚1 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9090268015861511 particle/accuracy_3_of_4‚1 particle/accuracy_6_of_6‚1 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.28388315439224243 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 particle/accuracy_3_of_6‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9290779829025269 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.33234432339668274 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_5‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.25 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 trainer/global_step‚10880 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9919553995132446 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.42592477798461914 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.019879063591361046 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4435740113258362 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.28579169511795044 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.18636946380138397 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.15625 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 particle/accuracy_1_of_1‚1 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.19166666269302368 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7393733263015747 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.026197347790002823 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5614035129547119 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3598039746284485 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18172457814216614 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9722222685813904 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.20192290842533112 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5066302418708801 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9629628658294678 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9921070337295532 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.993088960647583 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5429484844207764 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1865537315607071 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9722222089767456 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3873293399810791 +jet/accuracy_1_of_1‚0.45343026518821716 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4999999701976776 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3124999701976776 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.3333333134651184 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1418439745903015 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3149486184120178 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6345592141151428 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9864341616630554 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_4_of_6‚0.07692307978868484 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 particle/accuracy_2_of_4‚1 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.48765432834625244 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4716900885105133 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0026091919280588627 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.058497536927461624 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.44619905948638916 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.44908690452575684 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2516857087612152 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3980654776096344 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 (particle/f_score‚0.41064855456352234 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166667461395264 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.42365995049476624 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5151945352554321 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9905085563659668 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18523912131786346 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991071224212646 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 _runtime‚12949.580977215 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23914477229118347 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.017400285229086876 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ( Purity/*FRt3SRqqt2FBt/FBt_purity‚0.5 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6101189851760864 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5046296715736389 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ( Purity/1FRt3SRqqt2FBt/FBt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9985767006874084 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5203503966331482 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.45545434951782227 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4883982539176941 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.19109195470809937 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.08539094775915146 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 )jet/accuracy_1_of_3‚0.865862250328064 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1688888967037201 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.17733360826969147 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5278476476669312 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9388952851295471 7"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.45333331823349 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09849213063716888 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21353226900100708 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_6‚0.9807692170143127 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5625 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5594083666801453 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24875137209892273 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21540173888206482 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9895833134651184 jet/accuracy_5_of_6‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4643309414386749 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4503032863140106 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.532234251499176 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5359195470809937 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18172457814216614 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.5171536207199097 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.44510868191719055 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_6‚1 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08173885941505432 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23712612688541412 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.22972972691059113 +jet/accuracy_2_of_2‚0.10480955243110657 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9035088419914246 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02535771206021309 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9910793900489807 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4207688570022583 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0029320986941456795 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19866356253623962 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.026014110073447227 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.26218298077583313 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 *jet/accuracy_2_of_4‚0.3877880275249481 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09361612796783447 particle/accuracy_1_of_3‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.22268277406692505 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_5‚0.46444571018218994 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9889770746231079 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9911787509918213 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.4375 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.19624397158622742 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.24368685483932495 *jet/accuracy_2_of_3‚0.2548881471157074 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995235204696655 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.38997483253479004 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4937499463558197 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05729043111205101 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.43864747881889343 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5216204524040222 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3156357407569885 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2787356674671173 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 particle/accuracy_5_of_5‚1 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0885416641831398 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9892961978912354 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.44117647409439087 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4731040298938751 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6361575126647949 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1788337230682373 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5570860505104065 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.024766210466623306 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.14748375117778778 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.18040934205055237 6 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.131226047873497 jet/accuracy_4_of_5‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22839505970478058 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5778508186340332 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 jet/accuracy_7_of_7‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13683971762657166 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.008697137236595154 particle/accuracy_2_of_3‚1 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.25 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.17721278965473175 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9927732944488525 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.43691638112068176 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9711021780967712 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.08771929889917374 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.38249775767326355 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.559973955154419 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.982539713382721 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 +jet/accuracy_3_of_6‚0.18589743971824646 jet/accuracy_6_of_7‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 +particle/specificity‚0.9477952122688293 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 & Purity/1FRt1SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.05835579335689545 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.584801971912384 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9946901202201843 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8080903887748718 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4815220832824707 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7060967087745667 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1615992933511734 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18960444629192352 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.42365995049476624 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 *jet/accuracy_3_of_4‚0.0656547024846077 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.465717077255249 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9878621697425842 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9973360300064087 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.058423079550266266 particle/accuracy_1_of_5‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3925444185733795 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02649129554629326 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3977968692779541 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4773729145526886 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4901129901409149 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07574404776096344 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5447825193405151 ,particle/sensitivity‚0.33146172761917114 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9821428656578064 +jet/accuracy_2_of_6‚0.46794870495796204 jet/accuracy_3_of_7‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4839901030063629 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10376964509487152 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6013075113296509 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4784781336784363 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 6 Purity/1FRt3SRqqt*FBt/FBt_purity‚0.970588207244873 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4270833432674408 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15555214881896973 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_7‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.17617790400981903 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12339024990797043 particle/accuracy_3_of_5‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.40595242381095886 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 _step‚894 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5208333134651184 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3396291136741638 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9916458129882812 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.387275367975235 ,jet/accuracy_3_of_3‚0.027069954201579094 8"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.550000011920929 jet/accuracy_2_of_7‚0.5 particle/accuracy_7_of_7‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.030904285609722137 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9849275350570679 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3435862958431244 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_7‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.19345517456531525 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.011658296920359135 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 8"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.444444477558136 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5450134873390198 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.037553656846284866 particle/accuracy_2_of_6‚1 particle/accuracy_5_of_6‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.43290993571281433 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.34259262681007385 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.006945786066353321 particle/accuracy_3_of_3‚1 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.557504415512085 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4646887481212616 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 jet/accuracy_4_of_7‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4792725443840027 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 PurkW¬»ity/2FRt0SRqqt0FBt/FRt_purity‚0.19555702805519104 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.48580241203308105 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1637866348028183 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9940325617790222 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5385802388191223 particle/accuracy_1_of_4‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19010958075523376 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9399999976158142 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.19980792701244354 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 particle/accuracy_4_of_4‚1 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2648148238658905 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.35424837470054626 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9920826554298401 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5666666626930237 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25076594948768616 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285714030265808 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.48645833134651184 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.56048583984375 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5208333134651184 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 jet/accuracy_1_of_7‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.46364590525627136 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.44762709736824036 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5164728164672852 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 (particle/accuracy‚0.8515419363975525 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1778992861509323 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 jet/accuracy_5_of_7‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 *"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_6‚1 6 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.270049512386322 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.39832407236099243 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.491666704416275 +jet/accuracy_3_of_5‚0.07727853953838348 particle/accuracy_1_of_7‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 % _timestamp‚1.7885088918005483e+09 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23593658208847046 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3490511476993561 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19011422991752625 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5086538791656494 jet/accuracy_6_of_6‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 particle/accuracy_2_of_7‚1 *"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.5 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2986111044883728 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 epoch‚8 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.17260394990444183 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4280303120613098 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4151819348335266 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9930245876312256 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011920423246920109 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2202380895614624 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6537697911262512 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969|mÑ7h¯,ˆ¦ 8"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.550000011920929 jet/accuracy_2_of_7‚0.5 particle/accuracy_7_of_7‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.030904285609722137 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9849275350570679 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3435862958431244 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_7‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.19345517456531525 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.011658296920359135 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 8"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.444444477558136 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5450134873390198 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.037553656846284866 particle/accuracy_2_of_6‚1 particle/accuracy_5_of_6‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.43290993571281433 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.34259262681007385 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.006945786066353321 particle/accuracy_3_of_3‚1 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.557504415512085 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4646887481212616 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 jet/accuracy_4_of_7‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4792725443840027 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19555702805519104 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.48580241203308105 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1637866348028183 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9940325617790222 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5385802388191223 particle/accuracy_1_of_4‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19010958075523376 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9399999976158142 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.19980792701244354 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 particle/accuracy_4_of_4‚1 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2648148238658905 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.35424837470054626 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9920826554298401 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5666666626930237 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25076594948768616 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285714030265808 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.48645833134651184 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.56048583984375 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5208333134651184 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 jet/accuracy_1_of_7‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.46364590525627136 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.44762709736824036 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5164728164672852 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 (particle/accuracy‚0.8515419363975525 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1778992861509323 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 jet/accuracy_5_of_7‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 *"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_6‚1 6 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.270049512386322 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.39832407236099243 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.491666704416275 +jet/accuracy_3_of_5‚0.07727853953838348 particle/accuracy_1_of_7‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 % _timestamp‚1.7885088918005483e+09 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23593658208847046 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3490511476993561 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19011422991752625 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5086538791656494 jet/accuracy_6_of_6‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 particle/accuracy_2_of_7‚1 *"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.5 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2986111044883728 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 epoch‚8 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.17260394990444183 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4280303120613098 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4151819348335266 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9930245876312256 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011920423246920109 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2202380895614624 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6537697911262512 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2759627401828766 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4629629850387573 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25430285930633545 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4620808959007263 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.987500011920929 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.46021032333374023 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 )loss/val_total_loss‚39.98426035405653 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.44514259696006775 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1468253880739212 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24350202083587646 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9870368838310242 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4228779077529907 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.37021833658218384 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 particle/accuracy_1_of_2‚1 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9090268015861511 particle/accuracy_3_of_4‚1 particle/accuracy_6_of_6‚1 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.28388315439224243 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 particle/accuracy_3_of_6‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9290779829025269 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.33234432339668274 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_5‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.25 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 trainer/global_step‚10880 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9919553995132446 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.42592477798461914 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.019879063591361046 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4435740113258362 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.28579169511795044 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.18636946380138397 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.15625 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 particle/accuracy_1_of_1‚1 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.19166666269302368 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7393733263015747 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.026197347790002823 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5614035129547119 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3598039746284485 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18172457814216614 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9722222685813904 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.20192290842533112 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5066302418708801 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9629628658294678 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9921070337295532 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.993088960647583 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5429484844207764 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1865537315607071 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9722222089767456 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3873293399810791 +jet/accuracy_1_of_1‚0.45343026518821716 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4999999701976776 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3124999701976776 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.3333333134651184 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1418439745903015 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3149486184120178 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6345592141151428 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9864341616630554 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_4_of_6‚0.07692307978868484 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 particle/accuracy_2_of_4‚1 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.48765432834625244 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4716900885105133 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0026091919280588627 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.058497536927461624 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.44619905948638916 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.44908690452575684 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2516857087612152 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3980654776096344 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 (particle/f_score‚0.41064855456352234 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166667461395264 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.42365995049476624 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5151945352554321 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9905085563659668 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18523912131786346 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991071224212646 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 _runtime‚12949.580977215 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23914477229118347 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.017400285229086876 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ( Purity/*FRt3SRqqt2FBt/FBt_purity‚0.5 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6101189851760864 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5046296715736389 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ( Purity/1FRt3SRqqt2FBt/FBt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9985767006874084 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5203503966331482 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.45545434951782227 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4883982539176941 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.19109195470809937 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.08539094775915146 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 )jet/accuracy_1_of_3‚0.865862250328064 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1688888967037201 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.17733360826969147 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5278476476669312 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9388952851295471 7"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.45333331823349 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09849213063716888 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21353226900100708 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_6‚0.9807692170143127 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5625 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5594083666801453 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24875137209892273 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21540173888206482 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9895833134651184 jet/accuracy_5_of_6‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4643309414386749 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4503032863140106 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.532234251499176 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5359195470809937 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18172457814216614 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.5171536207199097 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.44510868191719055 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_6‚1 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08173885941505432 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23712612688541412 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.22972972691059113 +jet/accuracy_2_of_2‚0.10480955243110657 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9035088419914246 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02535771206021309 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9910793900489807 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4207688570022583 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0029320986941456795 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19866356253623962 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.026014110073447227 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.26218298077583313 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 *jet/accuracy_2_of_4‚0.3877880275249481 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09361612796783447 particle/accuracy_1_of_3‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.22268277406692505 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_5‚0.46444571018218994 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9889770746231079 ,&Purity/0FRt4SRqqt3FBt/event_propošE£iØ*rtion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9911787509918213 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.4375 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.19624397158622742 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.24368685483932495 *jet/accuracy_2_of_3‚0.2548881471157074 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995235204696655 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.38997483253479004 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4937499463558197 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05729043111205101 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.43864747881889343 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5216204524040222 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3156357407569885 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2787356674671173 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 particle/accuracy_5_of_5‚1 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0885416641831398 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9892961978912354 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.44117647409439087 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4731040298938751 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6361575126647949 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1788337230682373 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5570860505104065 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.024766210466623306 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.14748375117778778 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.18040934205055237 6 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.131226047873497 jet/accuracy_4_of_5‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22839505970478058 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5778508186340332 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 jet/accuracy_7_of_7‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13683971762657166 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.008697137236595154 particle/accuracy_2_of_3‚1 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.25 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.17721278965473175 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9927732944488525 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.43691638112068176 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9711021780967712 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.08771929889917374 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.38249775767326355 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.559973955154419 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.982539713382721 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 +jet/accuracy_3_of_6‚0.18589743971824646 jet/accuracy_6_of_7‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 +particle/specificity‚0.9477952122688293 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 & Purity/1FRt1SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.05835579335689545 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.584801971912384 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9946901202201843 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8080903887748718 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4815220832824707 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7060967087745667 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1615992933511734 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18960444629192352 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.42365995049476624 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 *jet/accuracy_3_of_4‚0.0656547024846077 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.465717077255249 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9878621697425842 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9973360300064087 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.058423079550266266 particle/accuracy_1_of_5‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3925444185733795 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02649129554629326 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3977968692779541 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4773729145526886 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4901129901409149 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07574404776096344 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5447825193405151 ,particle/sensitivity‚0.33146172761917114 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9821428656578064 +jet/accuracy_2_of_6‚0.46794870495796204 jet/accuracy_3_of_7‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4839901030063629 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10376964509487152 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6013075113296509 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4784781336784363 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 6 Purity/1FRt3SRqqt*FBt/FBt_purity‚0.970588207244873 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4270833432674408 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15555214881896973 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_7‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.17617790400981903 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12339024990797043 particle/accuracy_3_of_5‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.40595242381095886 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 _step‚894 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5208333134651184 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3396291136741638 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9916458129882812 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.387275367975235 ,jet/accuracy_3_of_3‚0.027069954201579094þ´!­ó“°,‚(:ˆ æõéÔûêÇm gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z±,‚(:ï æõéÔ€»n) disk./.usagePercent‚75.79393823024067 disk.rbd0.in‚0 disk.dm-0.in‚ 1684.23828125! disk.dm-0.out‚181275.91015625 disk.nvme0n1.in‚ 939.6640625 disk.sdb.in‚4.390625 network.recv‚ 4.190555e+06# disk.nvme1n1.out‚62383.65234375 disk.md0.in‚ 13.58203125 disk.md0.out‚ 1389.61328125 disk.sdb2.in‚4.390625 disk.md0p1.in‚ 13.58203125 disk.md0p1.out‚ 1389.61328125! disk.sdb2.out‚1444.5693359375! disk.sda2.out‚1444.5693359375$ memory_percent‚92.05622673034668 cpu‚25.21633747272146 disk.sda1.out‚0 disk.sda2.in‚ 9.19140625 disk.sda1.in‚0 disk.rbd0.out‚ 3.591796875 disk.sdb1.out‚0 disk.sda.out‚1444.5693359375$ disk./.usageGB‚2710.758358001709 disk.nvme1n1.in‚ 744.57421875# disk.nvme0n1.out‚118892.2578125# proc.memory.rssMB‚ 3012.59765625 network.sent‚ 2.9084014e+07 disk.rbd2.in‚0 disk.rbd2.out‚ 3.81640625 disk.sdb.out‚1444.5693359375( proc.memory.availableMB‚ 1301.5078125* proc.memory.percent‚18.387436866760254 proc.cpu.threads‚17 disk.sda.in‚ 9.19140625 disk.sdb1.in‚01AâI4²,/ _wandbruntime‚12970  _runtime‚12970Ìy´úȳ, _runtime‚12970.16678362 _step‚895 & lr-AdamW/pg1‚3.080375929466229e-05 & lr-AdamW/pg2‚3.080375929466229e-05 trainer/global_step‚10899 % _timestamp‚1.7885089123895817e+09n=éõÍ´,Ç % _timestamp‚1.7885089123895817e+09 _runtime‚12970.16678362 _step‚895 & lr-AdamW/pg1‚3.080375929466229e-05 & lr-AdamW/pg2‚3.080375929466229e-05 trainer/global_step‚10899ÿ ‘ÉR4µ,/ _wandbruntime‚12970  _runtime‚12970 yÇj (¶,†P YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚932943326642176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66281791 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _runtime‚12970.167845895 _step‚896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚920437826557440 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚889905233824112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 203219639 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚644 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1028437494987776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12457844039953 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 % _timestamp‚1.7885089123905423e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 228529684 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 203218993 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 136937202 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 504922765 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12505500084736 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12504590113280 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12505500084736 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 228529686 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 136937204 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12458752014957 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚920437808554496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 276419425 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1015384871921152 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚920437826557440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 276393081 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚902363985839069 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚932942398667776 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185Q•åŠÓ$·,‹P N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 203218993 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 136937202 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 504922765 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12505500084736 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12504590113280 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12505500084736 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 228529686 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 136937204 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12458752014957 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚920437808554496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 276419425 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1015384871921152 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚920437826557440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 276393081 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚902363985839069 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚932942398667776 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚932943326642176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66281791 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _runtime‚12970.167845895 _step‚896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚920437826557440 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚889905233824112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 203219639 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚644 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1028437494987776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12457844039953 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 % _timestamp‚1.7885089123905423e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 228529684 B0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368€ƒq 4¸,/ _wandbruntime‚12971  _runtime‚12971± '¹,N QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 136949269 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 228549844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 276417912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 _step‚897 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 504967756 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 276444282 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚920519427640320 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12506621558272 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1015475121565696 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13054016185856 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚902443195692527 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 203236775 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚933026067201536 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12505711573504 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚889984234379296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 239254528 trainer/global_step‚10899 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 276444282 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚920519427640320 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 504994128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 203237413 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 136949267 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚902444120707099 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12458961313231 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚920519445643264 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 228549846 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 228549846 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 136949269 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 228549844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 276417912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 _step‚897 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 504967756 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 276444282 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚920519427640320 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12506621558272 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1015475121565696 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13054016185856 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚902443195692527 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 203236775 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚933026067201536 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12505711573504 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚889984234379296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 239254528 trainer/global_step‚10899 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 276444282 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚920519427640320 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 504994128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 203237413 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 136949267 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚902444120707099 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12458961313231w‚CŒ4»,/ _wandbruntime‚12971  _runtime‚12971«×B¼_¼,Ù 1loss/FBt2/assignment_loss‚0.21839038757388707 /loss/FRt4/detection_loss‚3.9000630165849413 .loss/FBt3/detection_loss‚2.770842969417572 1loss/SRqqt2/assignment_loss‚1.576722155014674 1loss/SRqqt1/detection_loss‚2.2886359793799262 .loss/FBt4/detection_loss‚2.025514700346523 trainer/global_step‚10899 0loss/FRt4/assignment_loss‚2.2567757112639293 .loss/FRt2/detection_loss‚2.229930323885476 1loss/SRqqt3/detection_loss‚0.7951292970289404 epoch‚9 0loss/FRt1/assignment_loss‚2.6507223102781508 2loss/SRqqt3/assignment_loss‚0.8837806217981359 2loss/SRqqt4/assignment_loss‚1.9413853790611029 .loss/FRt3/detection_loss‚1.533040431983493 .loss/FBt2/detection_loss‚2.043886926025152 2loss/SRqqt1/assignment_loss‚0.8856482416923557 /loss/FBt4/assignment_loss‚0.277163623486634 0loss/FRt2/assignment_loss‚2.8391077531144973 3loss/FBt3/assignment_loss‚0.0020363492221662227 1loss/SRqqt2/detection_loss‚2.2566820563692036 /loss/FBt1/detection_loss‚0.7863730646204203 %loss/total_loss‚39.98448684626021 % _timestamp‚1.7885089135937085e+09 _runtime‚12971.370729988 0loss/FRt3/assignment_loss‚1.1612362916822787 1loss/FBt1/assignment_loss‚0.07193833492851098 /loss/FRt1/detection_loss‚1.9775426411383603 1loss/SRqqt4/detection_loss‚2.6119382803638778 _step‚8982JÓd½,Þ /loss/FBt1/detection_loss‚0.7863730646204203 %loss/total_loss‚39.98448684626021 % _timestamp‚1.7885089135937085e+09 _runtime‚12971.370729988 0loss/FRt3/assignment_loss‚1.1612362916822787 1loss/FBt1/assignment_loss‚0.07193833492851098 /loss/FRt1/detection_loss‚1.9775426411383603 1loss/SRqqt4/detection_loss‚2.6119382803638778 _step‚898 1loss/FBt2/assignment_loss‚0.21839038757388707 /loss/FRt4/detection_loss‚3.9000630165849413 .loss/FBt3/detection_loss‚2.770842969417572 1loss/SRqqt2/assignment_loss‚1.576722155014674 1loss/SRqqt1/detection_loss‚2.2886359793799262 .loss/FBt4/detection_loss‚2.025514700346523 trainer/global_step‚10899 0loss/FRt4/assignment_loss‚2.2567757112639293 .loss/FRt2/detection_loss‚2.229930323885476 1loss/SRqqt3/detection_loss‚0.7951292970289404 epoch‚9 0loss/FRt1/assignment_loss‚2.6507223102781508 2loss/SRqqt3/assignment_loss‚0.8837806217981359 2loss/SRqqt4/assignment_loss‚1.9413853790611029 .loss/FRt3/detection_loss‚1.533040431983493 .loss/FBt2/detection_loss‚2.043886926025152 2loss/SRqqt1/assignment_loss‚0.8856482416923557 /loss/FBt4/assignment_loss‚0.277163623486634 0loss/FRt2/assignment_loss‚2.8391077531144973 3loss/FBt3/assignment_loss‚0.0020363492221662227 1loss/SRqqt2/detection_loss‚2.2566820563692036‚ä•N“¾,‚(:ˆ õõéÔî°‘m gpu.0.gpu‚10.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.481' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.62025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÈêUèk¿,‚(:à õõéÔíÅço disk.sda1.out‚0) disk./.usagePercent‚75.79744714842641 disk.dm-0.out‚181276.3046875 disk.nvme0n1.in‚ 939.6640625 disk.nvme0n1.out‚ 118892.375 disk.sdb2.in‚4.390625 disk.sdb1.out‚0 network.recv‚ 4.194915e+06 disk.md0p1.in‚ 13.58203125 disk.dm-0.in‚ 1684.23828125 disk.md0.in‚ 13.58203125 disk.rbd0.in‚0$ memory_percent‚92.05632209777832* proc.memory.percent‚18.387436866760254 proc.cpu.threads‚17 disk.md0p1.out‚ 1391.203125 disk.sda2.in‚ 9.19140625 disk.sda2.out‚1446.212890625 disk.nvme1n1.in‚ 744.57421875 disk.rbd0.out‚ 3.591796875 disk.sdb1.in‚0 disk.md0.out‚ 1391.203125 disk.sda.in‚ 9.19140625 disk.sda.out‚1446.212890625( proc.memory.availableMB‚ 1301.4921875 cpu‚25.216120973225% disk./.usageGB‚2710.8838539123535 disk.sdb.out‚1446.212890625" disk.nvme1n1.out‚ 62383.9296875# proc.memory.rssMB‚ 3012.59765625 disk.sdb.in‚4.390625 disk.rbd2.in‚0 disk.rbd2.out‚ 3.81640625 disk.sdb2.out‚1446.212890625 disk.sda1.in‚0 network.sent‚ 2.9219139e+07ž!|“À,‚(:ˆ „öéÔÜøák gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚175.235' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.80875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/œ ¦qÁ,‚(:æ „öéÔˆÊÊl disk.sdb.out‚1448.1376953125 network.recv‚ 4.200954e+06% disk./.usageGB‚2709.7310180664062 disk.rbd2.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 1684.23828125& proc.memory.availableMB‚ 1301.09375 network.sent‚ 2.9313314e+07! disk.sdb2.out‚1448.1376953125 disk.sda1.out‚0 disk.nvme0n1.in‚ 939.6640625 disk.nvme0n1.out‚ 118892.375 disk.sdb2.in‚4.390625! disk.dm-0.out‚181614.33203125 disk.nvme1n1.in‚ 744.57421875# disk.nvme1n1.out‚62721.95703125 disk.md0.in‚ 13.58203125 disk.md0.out‚ 1393.078125 disk.sdb1.out‚0 disk.md0p1.in‚ 13.58203125 disk.md0p1.out‚ 1393.078125$ memory_percent‚92.05875396728516 disk.rbd2.out‚ 3.81640625 disk.rbd0.in‚0 disk.sdb1.in‚0) disk./.usagePercent‚75.76521337567443 disk.sda.in‚ 9.19140625 disk.sdb.in‚4.390625# proc.memory.rssMB‚ 3012.59765625* proc.memory.percent‚18.387436866760254 cpu‚25.21590395549206 disk.sda2.in‚ 9.19140625! disk.sda2.out‚1448.1376953125 disk.sda.out‚1448.1376953125 disk.rbd0.out‚ 3.591796875 proc.cpu.threads‚17“¼«’Â,‚(:‡ “öéÔ‡›µk gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.562' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.6405 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0κÌzzÃ,‚(:ï “öéÔ™ØÂl disk.sda1.in‚0 disk.md0p1.out‚ 1394.546875! disk.sdb.out‚1449.65771484375 disk.nvme0n1.in‚ 939.6640625$ disk.nvme0n1.out‚118892.43359375 disk.rbd2.in‚0 disk.dm-0.in‚ 1684.23828125# proc.memory.rssMB‚ 3012.59765625 network.recv‚ 4.205113e+06$ disk./.usageGB‚2709.552574157715 disk.sdb2.in‚4.390625" disk.sda2.out‚1449.65771484375$ memory_percent‚92.05889701843262( proc.memory.availableMB‚ 1301.0703125) disk./.usagePercent‚75.76022400930285 disk.sda.in‚ 9.19140625 disk.sda1.out‚0 disk.dm-0.out‚181748.5390625 disk.sda2.in‚ 9.19140625" disk.sdb2.out‚1449.65771484375 disk.rbd0.in‚0 disk.rbd0.out‚ 3.591796875 disk.md0.out‚ 1394.546875* proc.memory.percent‚18.387436866760254 disk.sdb1.in‚0 disk.sdb1.out‚0# disk.nvme1n1.out‚62856.10546875 proc.cpu.threads‚17! disk.sda.out‚1449.65771484375 disk.md0p1.in‚ 13.58203125 disk.nvme1n1.in‚ 744.57421875 cpu‚25.21567484369416 network.sent‚ 2.9319119e+07 disk.sdb.in‚4.390625 disk.md0.in‚ 13.58203125 disk.rbd2.out‚ 3.816406250éEù“Ä,‚(:ˆ ¢öéÔˆÓÒl gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.501' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.87525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08•ƒ–€Å,‚(:õ ¢öéÔÇ©¬m disk.sdb1.out‚0 disk.dm-0.in‚ 1684.23828125 disk.nvme1n1.in‚ 744.57421875) proc.memory.availableMB‚ 1301.10546875* proc.memory.percent‚18.387436866760254 proc.cpu.threads‚17) disk./.usagePercent‚75.75915153765236 disk.rbd0.in‚0 disk.md0p1.out‚ 1395.97265625" disk.sda2.out‚1451.14794921875$ disk.nvme0n1.out‚118892.43359375 network.recv‚ 4.209794e+06$ disk./.usageGB‚2709.514217376709 disk.sda1.out‚0 disk.sdb1.in‚0 disk.md0p1.in‚ 13.58203125 disk.dm-0.out‚181749.1796875" disk.sdb2.out‚1451.14794921875 disk.sda2.in‚ 9.19140625# disk.nvme1n1.out‚62856.74609375 disk.sda.in‚ 9.19140625$ memory_percent‚92.05868244171143! disk.sdb.out‚1451.14794921875 disk.md0.out‚ 1395.97265625 disk.rbd2.out‚ 3.81640625 cpu‚25.215461845273165 network.sent‚ 2.9327662e+07 disk.rbd0.out‚ 3.591796875 disk.nvme0n1.in‚ 939.6640625 disk.sdb2.in‚4.390625# proc.memory.rssMB‚ 3012.59765625 disk.sda1.in‚0 disk.sdb.in‚4.390625 disk.md0.in‚ 13.58203125 disk.rbd2.in‚0! disk.sda.out‚1451.14794921875õu/4Æ,/ _wandbruntime‚13022  _runtime‚13022Ç$qËÇ,Å ' lr-AdamW/pg2‚2.8294624029361222e-05 trainer/global_step‚10949 % _timestamp‚1.7885089651552784e+09 _runtime‚13022.932478539 _step‚899 ' lr-AdamW/pg1‚2.8294624029361222e-05†psÐÈ,Ê ' lr-AdamW/pg1‚2.8294624029361222e-05 ' lr-AdamW/pg2‚2.8294624029361222e-05 trainer/global_step‚10949 % _timestamp‚1.7885089651552784e+09 _runtime‚13022.932478539 _step‚899ƒ&èÓ4É,/ _wandbruntime‚13022  _runtime‚13022“R¶Ê,…P SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚924518780848640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66566468 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 277664327 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 204106920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚624 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1019896157143552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12561587509760 LBDeviceStatsMonitor.on_train_batch_s~3;ýUtart/allocation.small_pool.peak‚32006 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1033006992534528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 $ _timestamp‚1.788508965156348e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 507175667 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚937080368358400 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚924518762845696 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 229537686 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12514629270065 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚900 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13110835390976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 229537684 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚626 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263207936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 277637983 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚906370742821377 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚893856096511952 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 204107546 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚937079440384000 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1033007255742464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 _runtime‚13022.933662364 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚937079440384000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13111074658816 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1019896181083648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12561587509760 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 229537684 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚937080368358400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚893856113551312 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚924518762845696 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚924518780848640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12560677538304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66567092 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12513721295061 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12560677538304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 277637983 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 137540454 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 507202013 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 229537686 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 277664327 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 11953766408n=(Ë,ŠP =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 _runtime‚13022.933662364 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚937079440384000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13111074658816 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1019896181083648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12561587509760 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 229537684 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚937080368358400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚893856113551312 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚924518762845696 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚924518780848640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12560677538304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66567092 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12513721295061 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12560677538304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 277637983 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 137540454 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 507202013 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 229537686 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 277664327 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚924518780848640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66566468 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 277664327 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 204106920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚624 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1019896157143552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12561587509760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1033006992534528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 $ _timestamp‚1.788508965156348e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 507175667 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚937080368358400 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚924518762845696 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 229537686 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12514629270065 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚900 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13110835390976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 229537684 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚626 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263207936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 277637983 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚906370742821377 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚893856096511952 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 204107546 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚937079440384000 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1033007255742464„G¤o4Ì,/ _wandbruntime‚13023  _runtime‚13023¹^ÏÖ'Í,N bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13112235651584 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚924600381931520 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚924600399934464 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12562708983296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13112008979968 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12561798998528 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚937162180930048 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚924600399934464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66572198 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1033098391827968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 507247030 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 277689184 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1019986406788096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚937163108917760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 277689184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12514838568339 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 204125340 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 507247030 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚906449952674835 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚924600381931520 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 137552519 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 % _timestamp‚1.7885089662053328e+09 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 204124715 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1033098391827968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 507247030 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 277689184 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1019986406788096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.oÀl~Âín_train_batch_end/allocated_bytes.all.allocated‚937163108917760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 277689184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12514838568339 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 204125340 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 507247030 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚906449952674835 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚924600381931520 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 137552519 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 % _timestamp‚1.7885089662053328e+09 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 204124715 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 204997193 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 138143702 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12617674934784 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 241364992 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 509428569 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 230545686 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1024407466306048 trainer/global_step‚10999 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _runtime‚13079.171615734 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1037576985841664 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚897806976239152 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 509454915 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12569598550169 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 230545686 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚928599717136896 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 204997836 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚941216482100224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 230545684 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 265305088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 278909229 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66854132 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚910377499803685êY ‹˜"Ý,‹P D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 204997193 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 138143702 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12617674934784 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 241364992 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 509428569 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 230545686 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1024407466306048 trainer/global_step‚10999 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _runtime‚13079.171615734 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1037576985841664 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚897806976239152 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 509454915 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12569598550169 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 230545686 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚928599717136896 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 204997836 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚941216482100224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 230545684 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 26×Á#åy5305088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 278909229 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66854132 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚910377499803685 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚641 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13169519535616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13169278170624 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0ˆ#uM'sÞ,‚(:è ÞöéÔåýæl) disk./.usagePercent‚75.76946667771385 disk.md0.in‚ 13.58203125 disk.md0.out‚ 1402.3515625 proc.cpu.threads‚17 network.recv‚ 4.227814e+06 disk.rbd0.out‚ 3.603515625 disk.sda.in‚ 9.19140625! disk.dm-0.out‚182196.86328125! disk.sdb.out‚1457.79638671875 disk.sdb2.in‚4.390625 disk.md0p1.out‚ 1402.3515625 network.sent‚ 2.9450727e+07% disk./.usageGB‚2709.8831367492676 disk.sdb1.out‚0 disk.nvme1n1.in‚ 744.57421875* proc.memory.percent‚18.387436866760254 disk.dm-0.in‚ 1684.23828125" disk.sda2.out‚1457.79638671875 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sdb1.in‚0 disk.md0p1.in‚ 13.58203125 disk.sdb.in‚4.390625 disk.rbd2.out‚3.828125" disk.sdb2.out‚1457.79638671875! disk.sda.out‚1457.79638671875 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda2.in‚ 9.19140625 disk.nvme0n1.in‚ 939.6640625 disk.nvme0n1.out‚119338# disk.nvme1n1.out‚62858.86328125# proc.memory.rssMB‚ 3012.59765625 cpu‚25.21451573819535$ memory_percent‚92.06376075744629( proc.memory.availableMB‚ 1300.2734375sÍÁX“ß,‚(:ˆ ÞöéÔ¼—èl gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.637' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.15925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ2J²4à,/ _wandbruntime‚13080  _runtime‚13080)ߤ'á,ŽN VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚941300150633984 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12617886423552 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 230565846 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1024497668070400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚897886010873056 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 278934086 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚928681354225664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 509473560 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13170679895552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12571623798659 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1037668115002880 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12570715823447 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚928681354225664 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66859203 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1037668371906048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 278907716 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 138155767 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 230565846 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 509499932 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 509473560 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12618796408320 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 205014970 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚910456709657143 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12617886423552 _runtime‚13080.374284925 _step‚905 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚897885993833696 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚10999 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚910457634671715 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 278934086 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66859829 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 278907716 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 205015598 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212\ÏÃ'â,“N D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚10999 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚910457634671715 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 278934086 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66859829 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 278907716 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 205015598 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚941300150633984 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12617886423552 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 230565846 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1024497668070400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚897886010873056 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 278934086 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚928681354225664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 509473560 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13170679895552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12571623798659 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1037668115002880 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12570715823447 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚928681354225664 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66859203 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1037668371906048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 278907716 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 138155767 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 230565846 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 509499932 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 509473560 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12618796408320 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 205014970 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚910456709657143 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12617886423552 _runtime‚13080.374284925 _step‚905 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚897885993833696‰ x74ã,/ _wandbruntime‚13080  _runtime‚13080»ûŠZä,Ô epoch‚9 0loss/SRqqt1/detection_loss‚2.026542828083038 /loss/FBt2/detection_loss‚2.2396148294210434 /loss/FBt4/detection_loss‚1.7889948030312857 0loss/FRt3/assignment_loss‚1.0609452098608017 0loss/SRqqt2/detection_loss‚2.129961617614912 0loss/FBt3/assignment_loss‚0.8524874463037122 %loss/total_loss‚38.93439665201036 _runtime‚13080.375628331 2loss/SRqqt1/assignment_loss‚1.3792011007666587 /loss/FRt3/detection_loss‚1.5088250695417325 trainer/global_step‚10999 0loss/FRt1/assignment_loss‚1.9177063764358053 0loss/FRt4/assignment_loss‚1.9585949437958854 /loss/FRt1/detection_loss‚1.5267857414419639 1loss/SRqqt4/detection_loss‚2.3110305688210895 /loss/FBt1/detection_loss‚0.8705037387794462 _step‚906 0loss/FBt4/assignment_loss‚0.2031533537823028 /loss/FRt2/detection_loss‚2.3897650723345576 1loss/SRqqt3/detection_loss‚0.8995028391744443 -loss/FRt2/assignment_loss‚3.8117595911026 1loss/SRqqt4/assignment_loss‚1.237225139779704 0loss/FBt1/assignment_loss‚0.1730779395163001 /loss/FBt3/detection_loss‚2.9079766422510147 % _timestamp‚1.7885090225984483e+09 2loss/SRqqt2/assignment_loss‚1.5358788808402808 2loss/SRqqt3/assignment_loss‚0.9396053669849542 0loss/FBt2/assignment_loss‚0.2733534043131461 .loss/FRt4/detection_loss‚2.9919041480336875a U_å,Ù 0loss/SRqqt1/detection_loss‚2.026542828083038 /loss/FBt2/detection_loss‚2.2396148294210434 /loss/FBt4/detection_loss‚1.7889948030312857 0loss/FRt3/assignment_loss‚1.0609452098608017 0loss/SRqqt2/detection_loss‚2.129961617614912 0loss/FBt3/assignment_loss‚0.8524874463037122 %loss/total_loss‚38.93439665201036 _runtime‚13080.375628331 2loss/SRqqt1/assignment_loss‚1.3792011007666587 /loss/FRt3/detection_loss‚1.5088250695417325 trainer/global_step‚10999 0loss/FRt1/assignment_loss‚1.9177063764358053 0loss/FRt4/assignment_loss‚1.9585949437958854 /loss/FRt1/detection_loss‚1.5267857414419639 1loss/SRqqt4/detection_loss‚2.3110305688210895 /loss/FBt1/detection_loss‚0.8705037387794462 _step‚906 0loss/FBt4/assignment_loss‚0.2031533537823028 /loss/FRt2/detection_loss‚2.3897650723345576 1loss/SRqqt3/detection_loss‚0.8995028391744443 -loss/FRt2/assignment_loss‚3.8117595911026 1loss/SRqqt4/assignment_loss‚1.237225139779704 0loss/FBt1/assignment_loss‚0.1730779395163001 /loss/FBt3/detection_loss‚2.9079766422510147 % _timestamp‚1.7885090225984483e+09 2loss/SRqqt2/assignment_loss‚1.5358788808402808 2loss/SRqqt3/assignment_loss‚0.9396053669849542 0loss/FBt2/assignment_loss‚0.2733534043131461 .loss/FRt4/detection_loss‚2.991904148033687 epoch‚9ŠC‚‰zæ,‚(:ï íöéÔÕ‰n# proc.memory.rssMB‚ 3012.59765625 cpu‚25.214302880905567 disk.sda2.in‚ 9.19140625 disk.nvme1n1.in‚ 744.57421875 disk.sda.in‚ 9.19140625 disk.rbd0.in‚0 disk.rbd0.out‚ 3.666015625 disk.dm-0.in‚ 1684.23828125 network.recv‚ 4.232641e+06) disk./.usagePercent‚75.76535534084269$ disk./.usageGB‚2709.736095428467 disk.sdb.out‚1459.8505859375 disk.md0.out‚ 1404.33984375 disk.rbd2.out‚3.828125 disk.md0p1.in‚ 13.58203125* proc.memory.percent‚18.387436866760254# disk.nvme1n1.out‚62859.46484375 disk.sdb2.in‚4.390625 disk.md0p1.out‚ 1404.33984375 disk.sdb1.in‚0$ memory_percent‚92.06240177154541 disk.nvme0n1.in‚ 939.6640625 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17! disk.sdb2.out‚1459.8505859375 network.sent‚ 2.9546232e+07 disk.sdb.in‚4.390625$ disk.nvme0n1.out‚119433.12109375 disk.rbd2.in‚0 disk.sda.out‚1459.8505859375 disk.dm-0.out‚182292.5859375! disk.sda2.out‚1459.8505859375 disk.md0.in‚ 13.58203125 disk.sdb1.out‚0) proc.memory.availableMB‚ 1300.49609375Oókì“ç,‚(:ˆ íöéÔ˜ân gpu.0.gpu‚16.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚127.779' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.94475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢Ä\’è,‚(:‡ üöéÔ›¼–m gpu.0.gpu‚7.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.953' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.48825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨@Aw|é,‚(:ñ üöéÔÁôÊm disk.sda1.out‚0 disk.rbd0.out‚ 3.666015625 cpu‚25.214098791173917 disk.rbd2.out‚3.890625 disk.sdb1.out‚0% disk./.usageGB‚2709.8223266601562$ disk.nvme0n1.out‚119605.56640625! disk.nvme1n1.out‚ 62859.703125 disk.md0p1.in‚ 13.58203125 disk.md0p1.out‚ 1405.66796875 disk.sdb2.in‚4.390625 disk.sda.in‚ 9.19140625 disk.sda1.in‚0* proc.memory.percent‚18.387436866760254 network.sent‚ 2.964342e+07 disk.sdb.in‚4.390625 disk.nvme1n1.in‚ 744.57421875 disk.md0.out‚ 1405.66796875$ memory_percent‚92.06259250640869( disk./.usagePercent‚75.7677664021713 disk.dm-0.in‚ 1684.23828125! disk.dm-0.out‚182465.26953125! disk.sdb.out‚1461.23583984375 disk.sda2.in‚ 9.19140625 disk.md0.in‚ 13.58203125) proc.memory.availableMB‚ 1300.46484375 network.recv‚ 4.238287e+06# proc.memory.rssMB‚ 3012.59765625 disk.nvme0n1.in‚ 939.6640625 disk.rbd2.in‚0! disk.sda.out‚1461.23583984375 disk.rbd0.in‚0 disk.sdb1.in‚0 proc.cpu.threads‚17" disk.sda2.out‚1461.23583984375" disk.sdb2.out‚1461.23583984375°W¢‘ê,‚(:† ‹÷éÔƒèül gpu.0.gpu‚12.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚180.196' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.049 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶YŠlë,‚(:á ‹÷éÔÏ×m disk.sdb1.out‚0 disk.dm-0.in‚ 1684.23828125 disk.sdb.out‚ 1462.81640625# disk.nvme1n1.out‚62860.48046875 disk.md0.out‚ 1407.1953125$ memory_percent‚92.06328392028809% disk./.usageGB‚2710.1621284484863 disk.sda2.in‚ 9.19140625$ disk.nvme0n1.out‚119798.17578125# proc.memory.rssMB‚ 3012.59765625 cpu‚25.2138921842449 proc.cpu.threads‚17 network.sent‚ 2.9651887e+07 disk.rbd2.in‚0 disk.sdb2.in‚4.390625 disk.sda.in‚ 9.19140625 disk.sda1.out‚0 disk.md0p1.out‚ 1407.1953125 disk.dm-0.out‚ 182658.65625 disk.nvme1n1.in‚ 744.57421875 disk.rbd2.out‚3.890625 disk.md0p1.in‚ 13.58203125 disk.md0.in‚ 13.58203125( proc.memory.availableMB‚ 1300.3515625* proc.memory.percent‚18.387436866760254 network.recv‚ 4.243097e+06 disk.sdb1.in‚0 disk.sda2.out‚ 1462.81640625( disk./.usagePercent‚75.7772674023911 disk.sdb2.out‚ 1462.81640625 disk.sda.out‚ 1462.81640625 disk.sda1.in‚0 disk.rbd0.out‚ 3.7451171875 disk.sdb.in‚4.390625 disk.nvme0n1.in‚ 939.6640625 disk.rbd0.in‚0q²ìÅ4ì,/ _wandbruntime‚13137  _runtime‚13137çeq.Éí,à _runtime‚13137.743339733 _step‚907 & lr-AdamW/pg1‚2.358812851579884e-05 & lr-AdamW/pg2‚2.358812851579884e-05 trainer/global_step‚11049 % _timestamp‚1.7885090799660368e+09å éìÎî,È & lr-AdamW/pg2‚2.358812851579884e-05 trainer/global_step‚11049 % _timestamp‚1.7885090799660368e+09 _runtime‚13137.743339733 _step‚907 & lr-AdamW/pg1‚2.358812851579884e-05‹… Þ@4ï,/ _wandbruntime‚13137  _runtime‚13137ê6 Gð,…P HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12626383780281 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 511707817 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚945354451790848 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 231553684 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚932680689431040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚932680671428096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13227660224000 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚901757873005712 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 VGDeviceStatsMonit)û$Õ{!or.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 _runtime‚13137.744634839 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚932680671428096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 205888912 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1042146387812352 $ _timestamp‚1.788509079967211e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚945353523816448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚945354451790848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 280154131 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12673762359808 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 231553686 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1028918727588352 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67141958 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1042146642631680 trainer/global_step‚11049 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 280127787 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13227891103232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚914384256785993 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12625475805277 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 231553684 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 138746954 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚914383331771629 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 138746952 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 280127787 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 231553686 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚932680689431040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 280154131 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚945353523816448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚908 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 511681471 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12673762359808 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67141335 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚901757855966352 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0vÝOò(ñ,ŠP QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 _runtime‚13137.744634839 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚932680671428096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 205888912 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1042146387812352 $ _timestamp‚1.788509079967211e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚945353523816448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚945354451790848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 280154131 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12673762359808 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 231553686 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1028918727588352 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67141958 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1042146642631680 trainer/global_step‚11049 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 280127787 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13227891103232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚914384256785993 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12625475805277 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 231553684 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 138746954 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚914383331771629 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 138746952 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 280127787 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 231553686 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚932680689431040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 280154131 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚945353523816448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚908 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 511681471 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12673762359808 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67141335 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚901757855966352 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12626383780281 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 511707817 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚945354451790848 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 231553684 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚932680689431040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚932680671428096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13227660224000 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚901757873005712 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570ŒAå:4ò,/ _wandbruntime‚13138  _runtime‚13138›6l'ó,N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1029008953292800 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚932762290513920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚945436264362496 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1029008977232896 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 511752834 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 280178988 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚945437192350208 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 _step‚909 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚945437192350208 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13229058181632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12674883833344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 231573844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 205905942 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1042238035414528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 138759017 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚13138.789299549 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚901836873560896 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 138759019 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13228831510016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1042237784802816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 511752834 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 231573846 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚932762308516864 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚945437192350208 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 _step‚909 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚945437192350208 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13229058181632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12674883833344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 231573844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 205905942 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1042238035414528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 138759017 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚13138.789299549 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚901836873560896 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 138759019 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13228831510016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1042237784802816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 511752834 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 231573846 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚932762308516864 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67431628 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1046716186338816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 206781203 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 139350204 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 232561684 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67431001 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12729849784832 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚918390088753937 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12682261035389 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 281399033 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 281399033 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 232561686 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12728939813376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚936761625719296 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13286389727232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12728939813376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 281372689 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚627 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚918391013768301 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 206781832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚11099 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 139350202 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 232561684 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216199168 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13286173528064 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7885091386696484e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12729849784832 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚949490565532672 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚936761625719296 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚905708752732912 _runtime‚13196.447138517 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚949491493507072 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240139264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚936761643722240 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚949490565532672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12728939813376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 281372689 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚627 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚918391013768301 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 206781832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚11099 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 139350202 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 232561684 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216199168 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13286173528064 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7885091386696484e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12729849784832 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚949490565532672 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚936761625719296 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚905708752732912 _runtime‚13196.447138517 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚949491493507072 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240139264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚936761643722240 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚949490565532672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 TDDeviceStatsMonitor.on_train_batch_start/resþŒ6kÖ erved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67431628 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1046716186338816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 206781203 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 139350204 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 232561684 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67431001 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12729849784832 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚918390088753937 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12682261035389 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 281399033 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615*ÊWø4†-/ _wandbruntime‚13197  _runtime‚13197Ïѹ¼'‡-N MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 281423890 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1046807588276224 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12730061273600 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67436739 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _step‚913 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 281397520 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚905787770327456 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚936843244805120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 206799006 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚936843244805120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 513979364 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 232581846 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 139362267 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚13197.618028308 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12682470333663 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 trainer/global_step‚11099 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 % _timestamp‚1.7885091398396692e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 513979364 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚949574234066432 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 514005736 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ìƒ'ˆ-•N JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚936843262808064 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 281423890 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚905787753288096 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12730061273600 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67436739 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _step‚913 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 281397520 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚905787770327456 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚936843244805120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 206799006 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚936843244805120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 513979364 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 232581846 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 139362267 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚13197.618028308 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12682470333663 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 trainer/global_step‚11099 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 % _timestamp‚1.7885091398396692e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 513979364 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚949574234066432 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 514005736 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 281423890 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1046807588276224 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P4‰-/ _wandbruntime‚13197  _runtime‚13197©sCr\Š-Ö 0loss/FRt2/assignment_loss‚2.9953175454723593 1loss/FBt3/assignment_loss‚0.08263032700051554 1loss/FBt4/assignment_loss‚0.05522470038898642 /loss/FBt1/detection_loss‚0.6122097722463199 epoch‚9 trainer/global_step‚11099 2loss/SRqqt3/assignment_loss‚0.9737297088200104 0loss/FBt1/assignment_loss‚0.0718557531739244 /loss/SRqqt2/detection_loss‚2.61387117455403 /loss/FBt4/detection_loss‚2.6449031748554925 % _timestamp‚1.7885091398423471e+09 /loss/FRt1/assignment_loss‚2.093236716894003 1loss/SRqqt3/detection_loss‚0.7816961301748988 1loss/SRqqt4/detection_loss‚2.2393390825300505 .loss/FRt1/detection_loss‚1.719378318694138 .loss/FRt2/detection_loss‚2.141696080109295 /loss/FRt3/detection_loss‚1.6700265976458282 &loss/total_loss‚40.866986375533365 _step‚914 2loss/SRqqt2/assignment_loss‚1.2963195393482845 /loss/FRt4/detection_loss‚3.1948339210616217 /loss/FBt2/detection_loss‚2.3302009999752045 .loss/FBt3/detection_loss‚3.639358773827553 /loss/FRt4/assignment_loss‚2.649735653841937 1loss/FBt2/assignment_loss‚0.14552296924089206 _runtime‚13197.619484064 0loss/FRt3/assignment_loss‚1.5363940030841503 0loss/SRqqt1/assignment_loss‚1.37995574091162 2loss/SRqqt4/assignment_loss‚2.0169898212859123 1loss/SRqqt1/detection_loss‚1.9825598703963416ÁÑèa‹-Û 1loss/SRqqt1/detection_loss‚1.9825598703963416 0loss/FRt2/assignment_loss‚2.9953175454723593 1loss/FBt3/assignment_loss‚0.08263032700051554 1loss/FBt4/assignment_loss‚0.05522470038898642 /loss/FBt1/detection_loss‚0.6122097722463199 epoch‚9 trainer/global_step‚11099 2loss/SRqqt3/assignment_loss‚0.9737297088200104 0loss/FBt1/assignment_loss‚0.0718557531739244 /loss/SRqqt2/detection_loss‚2.61387117455403 /loss/FBt4/detection_loss‚2.6449031748554925 % _timestamp‚1.7885091398423471e+09 /loss/FRt1/assignment_loss‚2.093236716894003 1loss/SRqqt3/detection_loss‚0.7816961301748988 1loss/SRqqt4/detection_loss‚2.2393390825300505 .loss/FRt1/detection_loss‚1.719378318694138 .loss/FRt2/detection_loss‚2.141696080109295 /loss/FRt3/detection_loss‚1.6700265976458282 &loss/total_loss‚40.866986375533365 _step‚914 2loss/SRqqt2/assignment_loss‚1.2963195393482845 /loss/FRt4/detection_loss‚3.1948339210616217 /loss/FBt2/detection_loss‚2.3302009999752045 .loss/FBt3/detection_loss‚3.639358773827553 /loss/FRt4/assignment_loss‚2.649735653841937 1loss/FBt2/assignment_loss‚0.14552296924089206 _runtime‚13197.619484064 0loss/FRt3/assignment_loss‚1.5363940030841503 0loss/SRqqt1/assignment_loss‚1.37995574091162 2loss/SRqqt4/assignment_loss‚2.0169898212859123’¯=ó’Œ-‚(:‡ Ö÷éÔáø§n gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†I"Ðz-‚(:ï Ö÷éÔ‡°Ân disk.sdb1.in‚0 network.sent‚ 2.9797456e+07 network.recv‚ 4.273146e+06 disk.sda.in‚ 9.19140625 disk.rbd0.out‚ 3.87109375 disk.rbd2.in‚0) disk./.usagePercent‚75.79831568513947 disk.sdb1.out‚0 disk.nvme0n1.in‚ 939.6640625# proc.memory.rssMB‚ 3012.59765625 proc.cpu.threads‚17 disk.dm-0.in‚ 1684.23828125 disk.sda2.in‚ 9.19140625 disk.nvme1n1.in‚ 744.57421875 disk.md0.in‚ 13.58203125 disk.rbd2.out‚ 4.095703125$ memory_percent‚92.06485748291016& proc.memory.availableMB‚ 1300.09375 disk.md0p1.in‚ 13.58203125 disk.sdb.in‚4.390625! disk.sda2.out‚1471.4150390625$ disk.nvme0n1.out‚120590.36328125 disk.md0.out‚ 1415.50390625 disk.sdb2.in‚4.390625 cpu‚25.212864926492788 disk.sda.out‚1471.4150390625 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-0.out‚183451.8515625 disk.sdb.out‚1471.4150390625! disk.sdb2.out‚1471.4150390625% disk./.usageGB‚2710.9149169921875 disk.md0p1.out‚ 1415.50390625 disk.sda1.in‚0# disk.nvme1n1.out‚62861.48828125* proc.memory.percent‚18.3874368667602544‹d“Ž-‚(:ˆ å÷éÔͺåm gpu.0.gpu‚53.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.047' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.51175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“"O©|-‚(:ñ å÷éÔç¾ÿm! disk.sda.out‚1472.98388671875 network.sent‚ 2.9891629e+07" disk.sdb2.out‚1472.98388671875 network.recv‚ 4.278316e+06 disk.sda2.in‚ 9.19140625" disk.sda2.out‚1472.98388671875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 3.87109375! disk.sdb.out‚1472.98388671875 disk.nvme0n1.in‚ 939.6640625 disk.rbd2.out‚ 4.095703125# memory_percent‚92.0663833618164# proc.memory.rssMB‚ 3012.59765625 disk.sdb1.out‚0# disk.nvme1n1.out‚62861.68359375 disk.md0.in‚ 13.58203125 disk.sda1.in‚0 disk.md0p1.out‚ 1417.01171875 disk.dm-0.in‚ 1684.23828125$ disk.nvme0n1.out‚120719.66796875 disk.nvme1n1.in‚ 744.57421875 disk.md0.out‚ 1417.01171875 disk.sda.in‚ 9.19140625& proc.memory.availableMB‚ 1299.84375 proc.cpu.threads‚17) disk./.usagePercent‚75.79831632510266 disk.sdb1.in‚0 disk.md0p1.in‚ 13.58203125 disk.sdb.in‚4.390625 disk.sdb2.in‚4.390625* proc.memory.percent‚18.387436866760254 cpu‚25.212663634470264$ disk./.usageGB‚2710.914939880371 disk.dm-0.out‚183581.3515625 disk.rbd2.in‚0Ä7†P“-‚(:ˆ ô÷éÔ‹àýl gpu.0.gpu‚54.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.813' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.70325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…O’˜€‘-‚(:õ ô÷éÔ»ín disk.sda1.in‚0 disk.md0p1.in‚ 13.58203125 disk.nvme1n1.in‚ 744.57421875* proc.memory.percent‚18.387436866760254) disk./.usagePercent‚75.80530589643314 disk.md0.out‚ 1418.49609375 disk.sda.in‚ 9.19140625! disk.dm-0.out‚183768.05859375! disk.sda.out‚1474.52685546875 cpu‚25.212460610499615 network.recv‚ 4.282023e+06 disk.sda1.out‚0 disk.sdb1.in‚0$ disk.nvme0n1.out‚120906.19921875! disk.nvme1n1.out‚ 62861.859375 network.sent‚ 2.9898956e+07 disk.rbd0.in‚0 disk.sdb1.out‚0% disk./.usageGB‚2711.1649208068848 disk.rbd0.out‚ 3.87109375 disk.sda2.in‚ 9.19140625" disk.sda2.out‚1474.52685546875 disk.nvme0n1.in‚ 939.6640625 disk.sdb2.in‚4.390625 proc.cpu.threads‚17 disk.md0p1.out‚ 1418.49609375 disk.dm-0.in‚ 1684.23828125 disk.sdb.in‚4.390625 disk.md0.in‚ 13.58203125 disk.rbd2.out‚ 4.095703125" disk.sdb2.out‚1474.52685546875$ memory_percent‚92.06564426422119# proc.memory.rssMB‚ 3012.59765625! disk.sdb.out‚1474.52685546875 disk.rbd2.in‚0) proc.memory.availableMB‚ 1299.96484375f}‘’-‚(:† ƒøéÔÛ‡ºm gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚135.184' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^kÒ“-‚(:ô ƒøéÔð¡ÿm proc.cpu.threads‚17) disk./.usagePercent‚75.80530898958857 disk.sdb1.in‚0" disk.sda2.out‚1476.56201171875 disk.md0.in‚ 13.58203125 disk.rbd2.out‚ 4.095703125 disk.sdb2.in‚4.390625 disk.sda1.in‚0 disk.sda2.in‚ 9.19140625* proc.memory.percent‚18.387436866760254 disk.sdb1.out‚0! disk.dm-0.out‚183940.29296875 disk.nvme0n1.in‚ 939.6640625" disk.sdb2.out‚1476.56201171875 network.recv‚ 4.286164e+06! disk.sda.out‚1476.56201171875 disk.md0p1.in‚ 13.58203125 disk.nvme1n1.in‚ 744.57421875 disk.rbd2.in‚0 cpu‚25.212258298860558 disk.sda.in‚ 9.19140625 disk.rbd0.in‚0 disk.dm-0.in‚ 1684.23828125# disk.nvme0n1.out‚121078.2109375' proc.memory.availableMB‚ 1300.265625% disk./.usageGB‚2711.1650314331055 disk.sdb.in‚4.390625 disk.md0.out‚ 1420.46484375$ memory_percent‚92.06380844116211# proc.memory.rssMB‚ 3012.59765625 network.sent‚ 2.9902599e+07 disk.sda1.out‚0 disk.rbd0.out‚ 3.87109375 disk.md0p1.out‚ 1420.46484375! disk.sdb.out‚1476.56201171875# disk.nvme1n1.out‚62862.08203125’—ü&”-/ _wandbruntime‚13253  _rØx;Zuntime‚13253Êe„aË•-Å trainer/global_step‚11149 % _timestamp‚1.7885091958682415e+09 _runtime‚13253.645475649 _step‚915 ' lr-AdamW/pg1‚1.9300566134622822e-05 ' lr-AdamW/pg2‚1.9300566134622822e-05ÿ`¤ìЖ-Ê _step‚915 ' lr-AdamW/pg1‚1.9300566134622822e-05 ' lr-AdamW/pg2‚1.9300566134622822e-05 trainer/global_step‚11149 % _timestamp‚1.7885091958682415e+09 _runtime‚13253.645475649“ˆ…ý4—-/ _wandbruntime‚13253  _runtime‚13253yȉ (˜-†P JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚953627607248896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚940842598013440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216199168 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _step‚916 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12785937209856 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240139264 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚909659632460112 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12737230315493 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚953627607248896 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12785937209856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 282643935 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 282643935 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚940842598013440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 233569684 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚940842580010496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1037941298033152 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 233569684 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 233569686 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 207672931 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚940842580010496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 139953454 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1051285966867968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13344644894720 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚922396845736245 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 trainer/global_step‚11149úCQˆ(™-‹P KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚940842598013440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216199168 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _step‚916 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12785937209856 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240139264 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚909659632460112 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12737230315493 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚953627607248896 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12785937209856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 282643935 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 282643935 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚940842598013440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 233569684 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚940842580010496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1037941298033152 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 233569684 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 233569686 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 207672931 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚940842580010496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 139953454 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1051285966867968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13344644894720 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚922396845736245 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 trainer/global_step‚11149 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚953627607248896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784”ó$K4š-/ _wandbruntime‚13254  _runtime‚13254ª­Tƒ'›-N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 233589846 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1051377118112768 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12739255563983 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 282668792 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12786148698624 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 233589844 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 282668792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67725288 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 282642422 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 516232266 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 139965519 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚909738650054656 % _timestamp‚1.7885091968062608e+09 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 516232266 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 207691448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12787058683392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚940924217099264 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 282642422 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _runtime‚13254.584421141 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 233589844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚953710347794944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 139965517 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚922476980604067 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1038031547677696 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1038031523737600 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚953711275782656 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚940924199096320 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚909738633015296 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 233589846 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 207690805 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67725929 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 _step‚917 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 516258638 trainer/global_step‚11149 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12786148698624 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚953711275782656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12738347588771 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚922477905618639 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 282642422 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ?6DeviceSq¯Ëíž tatsMonitor.on_train_batch_end/segment.all.peak‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _runtime‚13254.584421141 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 233589844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚953710347794944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 139965517 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚922476980604067 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1038031547677696 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1038031523737600 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚953711275782656 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚940924199096320 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚909738633015296 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 233589846 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 207690805 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67725929 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 _step‚917 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 516258638 trainer/global_step‚11149 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12786148698624 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚953711275782656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12738347588771 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚922477905618639 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 516232266 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 139965519 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚909738650054656 % _timestamp‚1.7885091968062608e+09 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 516232266 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 207691448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12787058683392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2•Éô0P4-/ _wandbruntime‚13254  _runtime‚132547 T\ž-Ö .loss/FRt3/detection_loss‚1.423549488807718 0loss/SRqqt4/detection_loss‚2.459026713525095 /loss/FBt1/detection_loss‚0.5824894388796578 0loss/FRt1/assignment_loss‚1.9187085522545708 2loss/SRqqt1/assignment_loss‚1.4793975441716611 2loss/SRqqt2/assignment_loss‚1.8024819908594643 .loss/FBt3/detection_loss‚2.534290749579668 % _timestamp‚1.7885091968091385e+09 0loss/FRt4/assignment_loss‚3.1005295382605658 2loss/SRqqt3/assignment_loss‚0.8130004850543752 epoch‚9 trainer/global_step‚11149 0loss/FRt2/assignment_loss‚3.2751664990602536 1loss/FBt1/assignment_loss‚0.19067173605553378 0loss/SRqqt1/detection_loss‚1.765197766944766 1loss/FBt2/assignment_loss‚0.06114361569501246 1loss/FBt3/assignment_loss‚0.12164766880800926 1loss/FBt4/assignment_loss‚0.23465190857974735 /loss/FRt1/detection_loss‚1.8733784296452844 -loss/FRt2/detection_loss‚2.08020047150379 1loss/SRqqt2/detection_loss‚2.1786515878306494 _runtime‚13254.586124069 2loss/SRqqt4/assignment_loss‚1.5957717174483883 -loss/FBt2/detection_loss‚2.51099325021108 %loss/total_loss‚38.93471661273712 _step‚918 0loss/FRt3/assignment_loss‚1.0490518645806746 .loss/FRt4/detection_loss‚2.901636936046459 1loss/SRqqt3/detection_loss‚0.8089617902128878 /loss/FBt4/detection_loss‚2.1741168687218115¦æÜ-aŸ-Û .loss/FRt3/detection_loss‚1.423549488807718 0loss/SRqqt4/detection_loss‚2.459026713525095 /loss/FBt1/detection_loss‚0.5824894388796578 0loss/FRt1/assignment_loss‚1.9187085522545708 2loss/SRqqt1/assignment_loss‚1.4793975441716611 2loss/SRqqt2/assignment_loss‚1.8024819908594643 .loss/FBt3/detection_loss‚2.534290749579668 % _timestamp‚1.7885091968091385e+09 0loss/FRt4/assignment_loss‚3.1005295382605658 2loss/SRqqt3/assignment_loss‚0.8130004850543752 epoch‚9 trainer/global_step‚11149 0loss/FRt2/assignment_loss‚3.2751664990602536 1loss/FBt1/assignment_loss‚0.19067173605553378 0loss/SRqqt1/detection_loss‚1.765197766944766 1loss/FBt2/assignment_loss‚0.06114361569501246 1loss/FBt3/assignment_loss‚0.12164766880800926 1loss/FBt4/assignment_loss‚0.23465190857974735 /loss/FRt1/detection_loss‚1.8733784296452844 -loss/FRt2/detection_loss‚2.08020047150379 1loss/SRqqt2/detection_loss‚2.1786515878306494 _runtime‚13254.586124069 2loss/SRqqt4/assignment_loss‚1.5957717174483883 -loss/FBt2/detection_loss‚2.51099325021108 %loss/total_loss‚38.93471661273712 _step‚918 0loss/FRt3/assignment_loss‚1.0490518645806746 .loss/FRt4/detection_loss‚2.901636936046459 1loss/SRqqt3/detection_loss‚0.8089617902128878 /loss/FBt4/detection_loss‚2.1741168687218115–uXƒ“ -‚(:ˆ ’øéÔÿä±n gpu.0.gpu‚22.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚125.271' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.31775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01\C1{¡-‚(:ð ’øéÔùóín disk.sdb.out‚1478.0048828125$ disk./.usageGB‚2710.107608795166! disk.sdb2.out‚1478.0048828125 disk.sdb1.out‚0! disk.dm-0.out‚184108.91015625 disk.sda2.in‚ 9.19140625 disk.md0.out‚ 1421.84765625# proc.memory.rssMB‚ 3012.59765625 disk.md0p1.out‚ 1421.84765625 disk.sdb.in‚4.390625 disk.md0.in‚ 13.58203125 cpu‚25.21205281863095 disk.sda.out‚1478.0048828125 disk.sda1.out‚0 disk.nvme1n1.in‚ 744.57421875 proc.cpu.threads‚17 network.recv‚ 4.290788e+06) disk./.usagePercent‚75.77574301006602 disk.sda.in‚ 9.19140625 disk.rbd0.in‚0! disk.sda2.out‚1478.0048828125$ disk.nvme0n1.out‚121246.58984375 disk.rbd2.out‚ 4.095703125* proc.memory.percent‚18.387436866760254 network.sent‚ 2.9921747e+07 disk.rbd0.out‚ 3.87109375 disk.sdb1.in‚0 disk.md0p1.in‚ 13.58203125 disk.dm-0.in‚ 1684.23828125 disk.nvme0n1.in‚ 939.6640625) proc.memory.availableMB‚ 1299.67578125 disk.sdb2.in‚4.390625 disk.sda1.in‚0 disk.rbd2.in‚0$ memory_percent‚92.06740856170654" disk.nvme1n1.out‚ 62862.3203125vï—ù‘¢-‚(:† ¡øéÔú™‚m gpu.0.gpu‚20.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.676' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.419 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î <퀣-‚(:õ ¡øéÔn disk.nvme1n1.in‚ 744.57421875 disk.sdb2.in‚4.390625 disk.rbd0.out‚ 3.87109375 disk.md0p1.out‚ 1423.43359375 disk.dm-0.in‚ 1684.23828125 disk.nvme0n1.in‚ 939.6640625 disk.sda.in‚ 9.19140625 disk.sdb.in‚4.390625 disk.md0.in‚ 13.58203125 disk.rbd2.out‚ 4.095703125) disk./.usagePercent‚75.77574578323986# disk.nvme1n1.out‚62862.49609375! disk.sda.out‚1479.66455078125 disk.sdb1.out‚0 cpu‚25.211835378691855 network.recv‚ 4.295868e+06 network.sent‚ 3.0018254e+07# disk.nvme0n1.out‚121318.5859375 disk.sda1.in‚0$ memory_percent‚92.06478595733643) proc.memory.availableMB‚ 1300.10546875# proc.memory.rssMB‚ 3012.59765625 proc.cpu.threads‚17 disk.sda1.out‚0! disk.dm-0.out‚184181.08203125 disk.md0.out‚ 1423.43359375$ disk./.usageGB‚2710.107707977295 disk.sda2.in‚ 9.19140625" disk.sda2.out‚1479.66455078125 disk.md0p1.in‚ 13.58203125 disk.rbd2.in‚0" disk.sdb2.out‚1479.66455078125 disk.rbd0.in‚0 disk.sdb1.in‚0! disk.sdb.out‚1479.66455078125* proc.memory.percent‚18.387436866760254ß!ß“¤-‚(:ˆ °øéÔÅŒm gpu.0.gpu‚22.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.213' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.05325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Rõ{¥-‚(:ð °øéÔÌ©¤n disk.sdb2.in‚4.390625 disk.sdb.out‚1481.6728515625 disk.md0.in‚ 13.58203125$ memory_percent‚92.06404685974121 proc.cpu.threads‚17$ disk./.usageGB‚2710.232624053955 disk.rbd2.in‚0# proc.memory.rssMB‚ 3012.59765625 network.sent‚ 3.0022053e+07 network.recv‚ 4.300057e+06! disk.sdb2.out‚1481.6728515625 disk.rbd0.out‚ 3.87109375 disk.md0p1.in‚ 13.58203125 disk.dm-0.out‚184297.7265625( proc.memory.availableMB‚ 1300.2265625 cpu‚25.211614843568324 disk.sda.out‚1481.6728515625 disk.sda1.in‚0 disk.md0p1.out‚ 1425.38671875 disk.dm-0.in‚ 1684.23828125 disk.md0.out‚ 1425.38671875 disk.sdb1.in‚0 disk.sdb.in‚4.390625 disk.sda2.in‚ 9.19140625! disk.sda2.out‚1481.6728515625 disk.nvme1n1.in‚ 744.57421875* proc.memory.percent‚18.387436866760254) disk./.usagePercent‚75.77923848902472 disk.sda1.out‚0$ disk.nvme0n1.out‚121318.59765625# disk.nvme1n1.out‚62979.12890625 disk.sda.in‚ 9.19140625 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.rbd2.out‚ 4.095703125 disk.nvme0n1.in‚ 939.6640625±hæ¦-‚(:… ¿øéÔó‹m gpu.0.gpu‚21.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(ô†3~§-‚(:ó ¿øéÔÿ§n network.sent‚ 3.0030103e+07 network.recv‚ 4.304021e+06 disk.sda.in‚ 9.19140625! disk.sda.out‚1483.20849609375 disk.md0p1.in‚ 13.58203125 disk.sdb.in‚4.390625 disk.nvme0n1.in‚ 939.6640625 disk.rbd0.in‚0 cpu‚25.211414154509747 proc.cpu.threads‚17 disk.md0.in‚ 13.58203125% disk./.usageGB‚2710.0086059570312" disk.sdb2.out‚1483.20849609375 disk.md0p1.out‚ 1426.86328125 disk.dm-0.in‚ 1684.23828125" disk.sda2.out‚1483.20849609375 disk.nvme1n1.in‚ 744.57421875 disk.rbd2.out‚ 4.095703125) disk./.usagePercent‚75.77297484927587 disk.sdb2.in‚4.390625 disk.sda1.out‚0 disk.rbd0.out‚ 3.87109375 disk.sdb1.out‚0! disk.sdb.out‚1483.20849609375" disk.nvme0n1.out‚ 121427.578125$ memory_percent‚92.06399917602539 disk.sda1.in‚0 disk.md0.out‚ 1426.86328125 disk.rbd2.in‚0! disk.dm-0.out‚184407.09765625# proc.memory.rssMB‚ 3012.59765625 disk.sdb1.in‚0# disk.nvme1n1.out‚62979.51953125* proc.memory.percent‚18.387436866760254 disk.sda2.in‚ 9.19140625' proc.memory.availableMB‚ 1300.234375÷÷™~4¨-/ _wandbruntime‚13305  _runtime‚13305—ç@CÉ©-à & lr-AdamW/pg1‚1.731501543661702e-05 & lr-AdamW/pg2‚1.731501543661702e-05 trainer/global_step‚11199 % _timestamp‚1.7885092480293126e+09 _runtime‚13305.806566688 _step‚919Ž×' Ϊ-È & lr-AdamW/pg1‚1.731501543661702e-05 & lr-AdamW/pg2‚1.731501543661702e-05 trainer/global_step‚11199 % _timestamp‚1.7885092480293126e+09 _runtime‚13305.806566688 _step‚919—å«û4«-/ _wandbruntime‚13305  _runtime‚13305¯9½z (¬-†P YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚957765576939520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 283888837 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚926404527732917 _runtime‚13305.807812443 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 229653504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 234577686 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 283862493 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 140556704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 205713408 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 _step‚920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 234577686 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 208565632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚944923552304640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚944923534301696 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚926403602718553 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 518440177 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12793107570601 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12842024634880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1042452607195648 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚957764648965120 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68008315 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1042452583255552 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚957764648965120 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68008315 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1042452583255552 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚926404527732917 _runtime‚13305.807812443 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 229653504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 234577686 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 283862493 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 140556704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 205713408 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 _step‚920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 234577686 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 208565632˜›M4®-/ _wandbruntime‚13306  _runtime‚13306kYü'¯-N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 518485168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚13306.74678637 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 140568767 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚957848317498880 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 208583416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚957848317498880 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12843146108416 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 283913694 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 518511540 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 208582801 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 234597846 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 283887324 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1055947140397056 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 234597844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68014034 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12794224843879 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 283887324 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚945005153387520 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚615 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 234597846 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚945005171390464 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12842236123648 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 140568769 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1042542832900096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚957847389511168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚926483737586375 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 518511540 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12843146108416 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1055946885591040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚945005153387520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1042542808960000 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 % _timestamp‚1.7885092489690182e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚926484662600947 _step‚921 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚945005171390464 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 283913694 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13404307496960 trainer/global_step‚11199 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚913689529781856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784§râ'°-”N PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚13306.74678637 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 140568767 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚957848317498880 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 208583416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚957848317498880 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12843146108416 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 283913694 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 518511540 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 208582801 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 234597846 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 283887324 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1055947140397056 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 234597844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68014034 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12794224843879 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 283887324 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚945005153387520 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚615 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 234597846 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚945005171390464 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12842236123648 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 140568769 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1042542832900096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚957847389511168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚926483737586375 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 518511540 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12843146108416 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1055946885591040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚945005153387520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1042542808960000 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 % _timestamp‚1.7885092489690182e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚926484662600947 _step‚921 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚945005171390464 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 283913694 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13404307496960 trainer/global_step‚11199 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚913689529781856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 518485168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 235585684 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚924 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 235585684 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12898112059904 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 285107395 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 520693079 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚917561374875152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚961902618655744 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 235585686 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚68296176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68295542 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7885093051509895e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚11249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 141159952 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚13362.928573831 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216199168 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 209456130 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚917561391914512 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 285107395 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 520719425 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13461573057024 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚961901690681344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12848984825709 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚930411284715225 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚949004488592896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚949004488592896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 235585684 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚924 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 235585684 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12898112059904 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 285107395 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 520693079 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚917561374875152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚961902618655744 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 235585686 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚68296176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68295542 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7885093051509895e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚11249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 141159952 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚13362.928573831 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216199168 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 209456130 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚917561391914512 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 285107395 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 520719425 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13461573057024 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚961901690681344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12848984825709 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚930411284715225 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192œ>>•)4À-/ _wandbruntime‚13364  _runtime‚13364ÄÊ„'Á-N VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 141172019 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1060516866721280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 trainer/global_step‚11249 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12899233533440 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 285132226 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12850102098987 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13462513538560 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 235605844 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _step‚925 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 520764442 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 285158596 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1060516607720960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚930491419583255 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68301275 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 235605846 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12898323548672 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚949086125681664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12898323548672 % _timestamp‚1.7885093063432825e+09 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 141172017 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1047054094182400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235060224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 235605844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚930490494568683 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚917640409509056 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12851010074199 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 520738070 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 209473940 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 209473292 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 285158596 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 285132226 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚961985359215104 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 520764442 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12899233533440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚68301921 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1047054118122496 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 520738070 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚949086107678720S'‹Ž.Â-•N ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1060516866721280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 trainer/global_step‚11249 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12899233533440 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 285132226 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 YFDeviceStatsMonitor.on_train_batch_end/requested_byt&w?“í"es.small_pool.freed‚12850102098987 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13462513538560 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 235605844 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _step‚925 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 520764442 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 285158596 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1060516607720960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚930491419583255 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68301275 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 235605846 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12898323548672 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚949086125681664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12898323548672 % _timestamp‚1.7885093063432825e+09 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 141172017 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1047054094182400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235060224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 235605844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚930490494568683 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚917640409509056 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12851010074199 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 520738070 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 209473940 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 209473292 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 285158596 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 285132226 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚961985359215104 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 520764442 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12899233533440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚68301921 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1047054118122496 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 520738070 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚949086107678720 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 141172019Ê,§¬4Ã-/ _wandbruntime‚13364  _runtime‚13364+Ð]Ä-× 2loss/SRqqt1/assignment_loss‚1.2367386371562523 2loss/SRqqt3/assignment_loss‚0.7510620087307714 .loss/FBt2/detection_loss‚2.614116343168112 0loss/FRt4/assignment_loss‚2.6280718508519625 1loss/FBt1/assignment_loss‚0.09898466571521115 /loss/FRt3/detection_loss‚1.8889364606507544 .loss/FRt4/detection_loss‚3.310281392775084 1loss/SRqqt1/detection_loss‚1.7808072247675486 trainer/global_step‚11249 $ _timestamp‚1.788509306346823e+09 _step‚926 0loss/FRt3/assignment_loss‚1.2475540008185044 .loss/FRt2/detection_loss‚2.203970165594536 /loss/FBt1/detection_loss‚0.6168907913976373 /loss/FBt3/detection_loss‚1.8715832769870757 1loss/FBt4/assignment_loss‚0.37087557596334425 /loss/FRt1/detection_loss‚1.6726287573495937 1loss/SRqqt2/detection_loss‚2.0074488643942208 2loss/SRqqt2/assignment_loss‚1.5457872871694893 1loss/FBt2/assignment_loss‚0.27453123702644705 3loss/FBt3/assignment_loss‚0.0033962042912207835 1loss/SRqqt4/detection_loss‚2.5342121113990914 _runtime‚13364.124077835 /loss/FRt1/assignment_loss‚1.779275463687049 .loss/FRt2/assignment_loss‚2.78899956333871 .loss/FBt4/detection_loss‚2.089893877506256 2loss/SRqqt4/assignment_loss‚1.3932898405553966 epoch‚9 /loss/SRqqt3/detection_loss‚1.13987982404804 %loss/total_loss‚37.84921542534231 Ô ¨bÅ-Ü 2loss/SRqqt4/assignment_loss‚1.3932898405553966 epoch‚9 /loss/SRqqt3/detection_loss‚1.13987982404804 %loss/total_loss‚37.84921542534231 2loss/SRqqt1/assignment_loss‚1.2367386371562523 2loss/SRqqt3/assignment_loss‚0.7510620087307714 .loss/FBt2/detection_loss‚2.614116343168112 0loss/FRt4/assignment_loss‚2.6280718508519625 1loss/FBt1/assignment_loss‚0.09898466571521115 /loss/FRt3/detection_loss‚1.8889364606507544 .loss/FRt4/detection_loss‚3.310281392775084 1loss/SRqqt1/detection_loss‚1.7808072247675486 trainer/global_step‚11249 $ _timestamp‚1.788509306346823e+09 _step‚926 0loss/FRt3/assignment_loss‚1.2475540008185044 .loss/FRt2/detection_loss‚2.203970165594536 /loss/FBt1/detection_loss‚0.6168907913976373 /loss/FBt3/detection_loss‚1.8715832769870757 1loss/FBt4/assignment_loss‚0.37087557596334425 /loss/FRt1/detection_loss‚1.6726287573495937 1loss/SRqqt2/detection_loss‚2.0074488643942208 2loss/SRqqt2/assignment_loss‚1.5457872871694893 1loss/FBt2/assignment_loss‚0.27453123702644705 3loss/FBt3/assignment_loss‚0.0033962042912207835 1loss/SRqqt4/detection_loss‚2.5342121113990914 _runtime‚13364.124077835 /loss/FRt1/assignment_loss‚1.779275463687049 .loss/FRt2/assignment_loss‚2.78899956333871 .loss/FBt4/detection_loss‚2.089893877506256ž­`^“Æ-‚(:ˆ ûøéÔÒ©¸n gpu.0.gpu‚37.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.825' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.70625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Çò¦vÇ-‚(:ë ûøéÔåãÖn disk.sdb1.out‚0 disk.sdb.out‚1490.0537109375 disk.sda2.in‚ 9.19140625$ disk.nvme0n1.out‚121632.28515625! disk.sdb2.out‚1490.0537109375# proc.memory.rssMB‚ 3012.59765625 disk.md0.in‚ 13.58203125 disk.sda1.in‚0 disk.md0p1.in‚ 13.58203125 disk.dm-0.in‚ 1684.23828125 network.sent‚ 3.0166148e+07) disk./.usagePercent‚75.76953451381229 disk.nvme1n1.in‚ 744.57421875 disk.md0p1.out‚ 1433.4609375 disk.md0.out‚ 1433.4609375 disk.sdb.in‚4.390625# disk.nvme1n1.out‚63306.96484375 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.nvme0n1.in‚ 939.6640625 disk.sdb2.in‚4.390625 disk.sda.in‚ 9.19140625 disk.rbd0.out‚ 3.87109375 cpu‚25.210558711794782 proc.cpu.threads‚17! disk.sda2.out‚1490.0537109375 disk.sda1.out‚0 disk.dm-0.out‚ 184939.25* proc.memory.percent‚18.387436866760254 network.recv‚ 4.322135e+06% disk./.usageGB‚2709.8855628967285 disk.rbd2.out‚ 4.095703125 disk.sda.out‚1490.0537109375$ memory_percent‚92.06655025482178) proc.memory.availableMB‚ 1299.81640625½ap “È-‚(:ˆ ŠùéÔ®»²m gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.051' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.01275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®x£vÉ-‚(:ë ŠùéÔܪÎm) disk./.usagePercent‚75.77480770385985% disk./.usageGB‚2710.0741577148438$ disk.nvme0n1.out‚121633.21484375! disk.nvme1n1.out‚ 63371.140625 disk.rbd2.in‚0 disk.rbd2.out‚ 4.095703125 disk.sda.in‚ 9.19140625 disk.sda1.out‚0 disk.md0.in‚ 13.58203125! disk.sda2.out‚1491.5185546875( proc.memory.availableMB‚ 1299.6953125# proc.memory.rssMB‚ 3012.59765625 cpu‚25.210345966899187 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.sda2.in‚ 9.19140625$ memory_percent‚92.06728935241699 network.sent‚ 3.026345e+07! disk.sdb2.out‚1491.5185546875 disk.sda.out‚1491.5185546875* proc.memory.percent‚18.387436866760254 disk.md0.out‚ 1434.859375 disk.md0p1.out‚ 1434.859375 disk.nvme0n1.in‚ 939.6640625 disk.rbd0.in‚0 disk.rbd0.out‚ 3.87109375 disk.sdb1.in‚0 disk.dm-0.in‚ 1684.23828125! disk.dm-0.out‚185004.35546875 disk.sdb.in‚4.390625 disk.sdb.out‚1491.5185546875 disk.sda1.in‚0 disk.md0p1.in‚ 13.58203125 network.recv‚ 4.328265e+06 disk.nvme1n1.in‚ 744.57421875 disk.sdb2.in‚4.390625ruh‘Ê-‚(:† ™ùéÔŠ¼öl gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.676' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.419 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 šF±pË-‚(:å ™ùéÔ·«Òm disk.nvme0n1.in‚ 939.6640625 disk.nvme1n1.out‚ 63571.53125# proc.memory.rssMB‚ 3012.59765625 disk.sda.out‚1493.310546875 disk.rbd0.in‚0 disk.sda2.out‚1493.310546875 disk.sdb1.in‚0 disk.md0p1.out‚ 1436.59765625 disk.sdb.in‚4.390625 network.recv‚ 4.331694e+06 disk.rbd2.in‚0 disk.rbd2.out‚ 4.095703125 disk.md0p1.in‚ 13.58203125 disk.sda2.in‚ 9.19140625# disk.nvme0n1.out‚121633.5703125 disk.md0.out‚ 1436.59765625 proc.cpu.threads‚17 disk.nvme1n1.in‚ 744.57421875 cpu‚25.210147959436917) disk./.usagePercent‚75.77729118768977 disk.sda1.out‚0 disk.sdb1.out‚0 disk.sdb2.in‚4.390625 disk.dm-0.out‚185205.1015625 disk.md0.in‚ 13.58203125& proc.memory.availableMB‚ 1300.34375% disk./.usageGB‚2710.1629791259766 disk.sda.in‚ 9.19140625 disk.rbd0.out‚ 3.87109375 disk.dm-0.in‚ 1684.23828125 disk.sdb.out‚1493.310546875# memory_percent‚92.0633316040039* proc.memory.percent‚18.387436866760254 network.sent‚ 3.026882e+07 disk.sdb2.out‚1493.310546875 disk.sda1.in‚0ø°4’Ì-‚(:‡ ¨ùéÔ±ÿÁm gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚151.618' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.9045 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sxÒ xÍ-‚(:í ¨ùéÔ¥n disk.rbd2.out‚ 4.095703125 disk.dm-0.out‚185326.5859375 disk.sdb2.in‚4.390625" disk.sdb2.out‚1494.99853515625 disk.md0.out‚ 1438.20703125 disk.rbd2.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 1684.23828125! disk.sdb.out‚1494.99853515625 disk.sda2.in‚ 9.19140625 disk.nvme1n1.in‚ 744.57421875$ memory_percent‚92.06571578979492 proc.cpu.threads‚17 disk.sdb.in‚4.390625 network.sent‚ 3.027487e+07% disk./.usageGB‚2710.6649475097656 disk.nvme0n1.in‚ 939.6640625# disk.nvme0n1.out‚121633.5703125 disk.md0.in‚ 13.58203125 disk.sda1.out‚0# proc.memory.rssMB‚ 3012.59765625! disk.sda.out‚1494.99853515625 disk.rbd0.out‚ 3.87109375" disk.sda2.out‚1494.99853515625* proc.memory.percent‚18.387436866760254 disk.rbd0.in‚0' proc.memory.availableMB‚ 1299.953125 cpu‚25.2099480347554 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚ 13.58203125 disk.md0p1.out‚ 1438.20703125! disk.nvme1n1.out‚ 63693.015625 disk.sda.in‚ 9.19140625 network.recv‚ 4.336466e+06( disk./.usagePercent‚75.7913264337906Ï5°©4Î-/ _wandbruntime‚13421  _runtime‚13421&::ËÏ-Å _runtime‚13421.245392286 _step‚927 ' lr-AdamW/pg1‚1.3662443868484584e-05 ' lr-AdamW/pg2‚1.3662443868484584e-05 trainer/global_step‚11299 % _timestamp‚1.7885093634672036e+09Ø xÐÐ-Ê _runtime‚13421.245392286 _step‚927 ' lr-AdamW/pg1‚1.3662443868484584e-05 ' lr-AdamW/pg2‚1.3662443868484584e-05 trainer/global_step‚11299 % _timestamp‚1.7885093634672036e+09ŸÓg-@4Ñ-/ _wandbruntime‚13421  _runtime‚13421'*1 (Ò-†P \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 210346895 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 286352297 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚966039660371968 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚966038732397568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚953085442884096 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1064994969639424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12954199484928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚953085460887040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 522972327 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1064995220264448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 286378641 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13520042624000 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12904862080817 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚647 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 236593686 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 286352297 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚953085442884096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚953085460887040 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12953289513472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12905770055821 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 236593684 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚966039660371968 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1051475177640448 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 141763204 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚934418041697533 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 236593686 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 210347542 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚934417116683169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚68584338 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚966038732397568 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _runtime‚13421.245560116 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68583693ªš›£vÓ-‹P VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚68584338 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚966038732397568 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _runtime‚13421.245560116 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68583693 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 210346895 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 286352297 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚966039660371968 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚966038732397568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚953085442884096 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1064994969639424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12954199484928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚953085460887040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 522972327 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1064995220264448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 286378641 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13520042624000 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12904862080817 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_ba=%^"›tch_start/inactive_split.all.current‚647 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 236593686 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 286352297 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚953085442884096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚953085460887040 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12953289513472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12905770055821 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 236593684 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚966039660371968 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1051475177640448 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 141763204 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚934418041697533 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 236593686 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 210347542 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚934417116683169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 ×…ö4Ô-/ _wandbruntime‚13422  _runtime‚13422 ͨ'Õ-N EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚13422.22600436 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 523017344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚642 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 trainer/global_step‚11299 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 210364710 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚934497251550991 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12905979354095 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚953167079972864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1065086616867840 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1051565379404800 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 286403498 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13521213522944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 522990972 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12906887329307 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 236613846 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 286377128 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12954410973696 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 236613844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚966121472943616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 141775267 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 _step‚929 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚966122400931328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12955320958464 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12955320958464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1065086364158976 % _timestamp‚1.7885093644481258e+09 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 236613846 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1051565403344896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚921591289236256 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 236613844 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12954410973696 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 523017344 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚921591272196896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0?àñ`'Ö-”N SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 286403498 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚68590083 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚953167061969920 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚13422.22600436 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 523017344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚642 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 trainer/global_step‚11299 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 210364710 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚934497251550991 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12905979354095 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚953167079972864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1065086616867840 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1051565379404800 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 286403498 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13521213522944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 522990972 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12906887329307 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 236613846 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 286377128 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12954410973696 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 236613844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚966121472943616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 141775267 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 _step‚929 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚966122400931328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12955320958464 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12955320958464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1065086364158976 % _timestamp‚1.7885093644481258e+09 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 236613846 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1051565403344896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚921591289236256 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 236613844 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12954410973696 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 523017344 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚921591272196896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 237601684 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1055986438922752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 287597199 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 287623543 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚970175774113792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _step‚932 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚970175774113792 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚957166415178240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 525198883 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13578515232256 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚938423873665477 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚925463151368912 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12960739335925 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚970176702088192 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚957166397175296 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13010286909952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13578307421696 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚938424798679841 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 142366454 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 142366452 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 207810560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 237601684 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13009376938496 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚957166397175296 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚626 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚68872224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68871600 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 287623543 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1069564978095104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13010286909952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚11349 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚925463134329552 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 237601686 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚957166415178240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 _runtime‚13479.831735477 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 211238678 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 237601686 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 231750656 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13009376938496 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 211238052 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1069564746344448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1055986462862848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640=˜[(ç-‹P YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚970176702088192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 525225229 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 287597199 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 _runtime‚13479.831735477 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 211238678 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 237601686 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 231750656 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13009376938496 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 211238052 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1069564746344448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1055986462862848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 237601684 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1055986438922752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 287597199 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 287623543 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚970175774113792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _step‚932 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚970175774113792 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚957166415178240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 525198883 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13578515232256 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚938423873665477 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚925463151368912 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12960739335925 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚970176702088192 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚957166397175296 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13010286909952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13578307421696 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚938424798679841 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 142366454 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 142366452 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 207810560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 237601684 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13009376938496 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚957166397175296 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚626 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚68872224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68871600 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 287623543 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1069564978095104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13010286909952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚11349 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚925463134329552 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 237601686 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚957166415178240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 J4è-/ _wandbruntime‚13481  _runtime‚13481Ï¿Ã#é-N SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 237621844 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚925542168963456 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13010498398720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _step‚933 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚957248016261120 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 237621846 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1056076664627200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚970258514659840 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 287648400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68877323 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13579698491904 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚635 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 142378519 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885094232294128e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 525270246 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13010498398720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚970259442647552 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1069656387059200 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1069656130156032 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 525270246 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 287622030 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚68877956 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 trainer/global_step‚11349 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13011408383488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 287648400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 525243874 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚633 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13579465528832 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 525243874 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚938504008533299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 211255840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚938504933547871 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12962764584415 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885094232294128e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 525270246 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13010498398720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚970259442647552 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1069656387059200 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1069656130156032 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 525270246 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 287622030 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚68877956 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 trainer/global_step‚11349 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13011408383488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 287648400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 525243874 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚633 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13579465528832 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 525243874 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚938504008533299 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 211255840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚938504933547871 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12962764584415 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚925542168963456 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13010498398720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _step‚933 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚957248016261120 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 237621846 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1056076664627200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144¥nƒ»4ë-/ _wandbruntime‚13481  _runtime‚13481ìÀ\ì-Ö 2loss/SRqqt3/assignment_loss‚0.7026682626749352 2loss/SRqqt4/assignment_loss‚1.5722217275010002 1loss/FBt2/assignment_loss‚0.19650668425369985 epoch‚9 /loss/FRt1/assignment_loss‚2.775850745042165 0loss/FRt2/assignment_loss‚3.2967577183788475 2loss/SRqqt1/assignment_loss‚1.5991106217815763 .loss/FRt4/detection_loss‚2.662505158356258 1loss/SRqqt1/detection_loss‚2.5908022267477855 /loss/FBt3/detection_loss‚1.9750716090202332 $ _timestamp‚1.788509423231466e+09 0loss/FRt3/assignment_loss‚1.3960983834024203 /loss/FRt4/assignment_loss‚2.724143273489816 0loss/SRqqt2/detection_loss‚2.141844868659973 _step‚934 1loss/FBt1/assignment_loss‚0.11012267853245229 0loss/FBt3/assignment_loss‚0.4868279682090765 _runtime‚13481.00853531 1loss/SRqqt3/detection_loss‚0.8498177246051276 .loss/FBt2/detection_loss‚2.117485757668813 %loss/total_loss‚39.94806280627988 trainer/global_step‚11349 /loss/FRt2/detection_loss‚2.3249966423027217 2loss/SRqqt2/assignment_loss‚1.5390727928569239 /loss/FRt1/detection_loss‚1.8545802837806857 /loss/FRt3/detection_loss‚1.6186413275614633 /loss/FBt1/detection_loss‚0.6982347209316989 1loss/FBt4/assignment_loss‚0.11636466052013103 0loss/SRqqt4/detection_loss‚2.291255037630758 /loss/FBt4/detection_loss‚2.3070819323713128Žµš—aí-Û /loss/FBt3/detection_loss‚1.9750716090202332 $ _timestamp‚1.788509423231466e+09 0loss/FRt3/assignment_loss‚1.3960983834024203 /loss/FRt4/assignment_loss‚2.724143273489816 0loss/SRqqt2/detection_loss‚2.141844868659973 _step‚934 1loss/FBt1/assignment_loss‚0.11012267853245229 0loss/FBt3/assignment_loss‚0.4868279682090765 _runtime‚13481.00853531 1loss/SRqqt3/detection_loss‚0.8498177246051276 .loss/FBt2/detection_loss‚2.117485757668813 %loss/total_loss‚39.94806280627988 trainer/global_step‚11349 /loss/FRt2/detection_loss‚2.3249966423027217 2loss/SRqqt2/assignment_loss‚1.5390727928569239 /loss/FRt1/detection_loss‚1.8545802837806857 /loss/FRt3/detection_loss‚1.6186413275614633 /loss/FBt1/detection_loss‚0.6982347209316989 1loss/FBt4/assignment_loss‚0.11636466052013103 0loss/SRqqt4/detection_loss‚2.291255037630758 /loss/FBt4/detection_loss‚2.3070819323713128 2loss/SRqqt3/assignment_loss‚0.7026682626749352 2loss/SRqqt4/assignment_loss‚1.5722217275010002 1loss/FBt2/assignment_loss‚0.19650668425369985 epoch‚9 /loss/FRt1/assignment_loss‚2.775850745042165 0loss/FRt2/assignment_loss‚3.2967577183788475 2loss/SRqqt1/assignment_loss‚1.5991106217815763 .loss/FRt4/detection_loss‚2.662505158356258 1loss/SRqqt1/detection_loss‚2.5908022267477855¦¾ aÈ~î-‚(:ó óùéÔ·¸²o! disk.nvme1n1.out‚ 64543.859375 disk.rbd2.out‚ 4.095703125 disk.md0p1.in‚ 13.58203125 network.recv‚ 4.357949e+06 disk.sdb2.in‚4.390625" disk.sdb2.out‚1504.92333984375! disk.sda.out‚1504.92333984375 disk.sda1.out‚0 cpu‚25.20896326778151 proc.cpu.threads‚17 disk.sda2.in‚ 9.19140625$ disk.nvme0n1.out‚121634.12109375 disk.nvme1n1.in‚ 744.57421875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 1684.23828125 disk.rbd2.in‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 1300.82421875# proc.memory.rssMB‚ 3012.59765625* proc.memory.percent‚18.387436866760254 disk.rbd0.out‚ 3.87109375 disk.md0p1.out‚ 1447.79296875$ disk./.usageGB‚2710.475654602051! disk.sdb.out‚1504.92333984375 disk.md0.in‚ 13.58203125 network.sent‚ 3.0415608e+07) disk./.usagePercent‚75.78603372486566! disk.dm-0.out‚186177.98046875$ memory_percent‚92.06039905548096 disk.sdb.in‚4.390625" disk.sda2.out‚1504.92333984375 disk.md0.out‚ 1447.79296875 disk.sda.in‚ 9.19140625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 939.6640625Òߵʒï-‚(:‡ óùéÔø˜Äo gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚151.618' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.9045 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®¶“ð-‚(:ˆ ‚úéÔœœm gpu.0.gpu‚57.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚125.271' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.31775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]ìó~ñ-‚(:ó ‚úéÔÖ¶Ám disk.sda1.in‚0 disk.rbd0.in‚0 disk.md0.out‚ 1449.7265625) disk./.usagePercent‚75.79302638935155$ disk.nvme0n1.out‚121634.12109375" disk.nvme1n1.out‚ 64731.9609375 disk.rbd2.in‚0) proc.memory.availableMB‚ 1302.87890625" disk.sdb2.out‚1506.91943359375 disk.rbd2.out‚ 4.095703125 proc.cpu.threads‚17 network.sent‚ 3.0510756e+07 disk.sdb2.in‚4.390625 disk.sda1.out‚0 disk.rbd0.out‚ 3.87109375 disk.nvme1n1.in‚ 744.57421875# proc.memory.rssMB‚ 3012.59765625$ disk./.usageGB‚2710.725746154785 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sdb.in‚4.390625 disk.sda2.in‚ 9.19140625! disk.sda.out‚1506.91943359375 disk.dm-0.in‚ 1684.23828125" disk.sda2.out‚1506.91943359375 disk.nvme0n1.in‚ 939.6640625 network.recv‚ 4.363979e+06 disk.md0p1.in‚ 13.58203125 disk.md0p1.out‚ 1449.7265625! disk.sdb.out‚1506.91943359375 disk.md0.in‚ 13.58203125$ memory_percent‚92.04785823822021* proc.memory.percent‚18.387436866760254 disk.sda.in‚ 9.19140625! disk.dm-0.out‚186366.08203125 cpu‚25.208757343867088á1ض‘ò-‚(:† ‘úéÔªäm gpu.0.gpu‚50.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚175.512' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@¼*ró-‚(:ç ‘úéÔ€üüm disk.rbd2.out‚ 4.095703125 disk.sda1.out‚0# proc.memory.rssMB‚ 3012.59765625 cpu‚25.208561209224644( disk./.usagePercent‚75.7757455699188$ disk.nvme0n1.out‚121634.17578125 disk.sda.in‚ 9.19140625 disk.sdb1.in‚0% disk./.usageGB‚2710.1077003479004 disk.sda2.out‚1508.857421875 disk.nvme1n1.in‚ 744.57421875 disk.rbd2.in‚0 disk.sdb2.in‚4.390625 disk.rbd0.out‚ 3.87109375 disk.md0p1.out‚ 1451.59765625 network.sent‚ 3.0516033e+07 disk.sdb.in‚4.390625 disk.sdb2.out‚1508.857421875 disk.sda.out‚1508.857421875 disk.dm-0.in‚ 1684.23828125* proc.memory.percent‚18.387436866760254 network.recv‚ 4.36724e+06 disk.nvme0n1.in‚ 939.6640625# disk.nvme1n1.out‚64956.48828125 disk.md0.out‚ 1451.59765625 disk.sda1.in‚0$ memory_percent‚92.04940795898438 disk.sda2.in‚ 9.19140625 disk.md0.in‚ 13.58203125 disk.dm-0.out‚186590.6640625 disk.sdb.out‚1508.857421875 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚ 13.58203125$ proc.memory.availableMB‚1302.625 proc.cpu.threads‚17³“ô-‚(:ˆ  úéÔÐðül gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.333' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.33325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`Ìh{õ-‚(:ð  úéÔ¨é‡n disk.rbd2.out‚ 4.095703125 disk.sda1.out‚0$ disk.nvme0n1.out‚121634.17578125! disk.sdb2.out‚1510.1748046875 disk.nvme1n1.in‚ 744.57421875 disk.rbd2.in‚0 disk.sdb2.in‚4.390625 disk.sda.in‚ 9.19140625! disk.dm-0.out‚186707.30859375 disk.sda2.in‚ 9.19140625! disk.sda2.out‚1510.1748046875* proc.memory.percent‚18.387436866760254 disk.md0.in‚ 13.58203125 disk.md0.out‚ 1452.85546875 disk.dm-0.in‚ 1684.23828125 disk.sdb.in‚4.390625 cpu‚25.208348532979073 proc.cpu.threads‚17$ disk./.usageGB‚2710.232711791992 disk.sdb1.out‚0 disk.md0p1.out‚ 1452.85546875 network.sent‚ 3.0524311e+07 disk.nvme0n1.in‚ 939.6640625 disk.sda1.in‚0 disk.rbd0.out‚ 3.87109375 disk.sdb1.in‚0( proc.memory.availableMB‚ 1303.0703125) disk./.usagePercent‚75.77924094221696" disk.nvme1n1.out‚ 65073.1328125 disk.rbd0.in‚0 disk.md0p1.in‚ 13.58203125 disk.sdb.out‚1510.1748046875# proc.memory.rssMB‚ 3012.59765625 network.recv‚ 4.371491e+06 disk.sda.out‚1510.1748046875$ memory_percent‚92.04668998718262$²{4ö-/ _wandbruntime‚13532  _runtime‚13532'öí¤Ë÷-Å ' lr-AdamW/pg1‚1.0437078992880894e-05 ' lr-AdamW/pg2‚1.0437078992880894e-05 trainer/global_step‚11399 % _timestamp‚1.7885094750648196e+09 _runtime‚13532.842141456 _step‚935ȯ EÐø-Ê trainer/global_step‚11399 % _timestamp‚1.7885094750648196e+09 _runtime‚13532.842141456 _step‚935 ' lr-AdamW/pg1‚1.0437078992880894e-05 ' lr-AdamW/pg2‚1.0437078992880894e-05§Ãáú4ù-/ _wandbruntime‚13532  _runtime‚13532h_€c (ú-†P MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚961247369469440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚636 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚961247351466496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13066374334976 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 212130382 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1060497748085248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚929414014056752 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13065464363520 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1074134386199552 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚69160044 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 238609686 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 288842101 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚936 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _runtime‚13532.843427922 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 212129746 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚634 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚974312815830016 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13066374334976 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 288868445 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚942431555662149 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13636895030784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 527451785 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13017524566037 % _timestamp‚1.7885094750660534e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 238609686 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚942430630647785 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚11399 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 288868445 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 527478131 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 238609684 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚974313743804416 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 142969702 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚974313743804416 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚974312815830016 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13636662054400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚69160678 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13017524566037 % _timestamp‚1.7885094750660534e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 238609686 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚942430630647785 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚11399 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batcY@‰ló h_start/active.small_pool.allocated‚ 288868445 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 527478131 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 238609684 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚974313743804416 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 142969702 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚974313743804416 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚974312815830016 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13636662054400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚69160678 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚929414014056752 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13065464363520 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1074134386199552 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚69160044 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 238609686 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 288842101 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚936 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _runtime‚13532.843427922 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 212129746 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚634 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚974312815830016 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13066374334976 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 288868445 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚942431555662149 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13636895030784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 527451785 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 212147545 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚929493048690656 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 238629846 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚974396484363776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚621 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 288893302 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 212148166 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 288893302 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 238629844 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 _step‚937 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13066585823744 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚942511690530179 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚961328988555264 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 142981767 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13017733864311 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚942510765515607 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 527523148 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13066585823744 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13637832035328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 142981769 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 527496776 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 238629844 _runtime‚13533.807526918 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚974396484363776 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚619 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69165778 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 527523148 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185ìDšz'þ-•N >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 212147545 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚929493048690656 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 238629846 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚974396484363776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚621 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 288893302 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 212148166 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 288893302 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 238629844 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 _step‚937 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13066585823744 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚942511690530179 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚961328988555264 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 142981767 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13017733864311 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚942510765515607 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 527523148 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13066585823744 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13637832035328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 142981769 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 527496776 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 238629844 _runtime‚13533.807526918 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚974396484363776 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚619 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69165778 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 527523148 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚11399 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚978449857546240 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 529731033 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 213023133 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚946438312644457 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13121551788544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚965328323760640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚635 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1065009033307648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 529704687 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 290087003 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13122461760000 trainer/global_step‚11449 % _timestamp‚1.7885095300578544e+09 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13695252089344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚933364910823312 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚978449857546240 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 239617686 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚933364893783952 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 239617684 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 143572954 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1078704514179072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 143572952 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1065009009367552 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13072493846141 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 239617686 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 _runtime‚13587.835176413 _step‚940 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13073401821145 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚965328305757696 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 290087003 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13695480871424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640h1÷£(.‹P [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1078704261456896 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚978449857546240 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 529731033 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 213023133 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚946438312644457 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13121551788544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚965328323760640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚635 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1065009033307648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 529704687 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 290087003 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13122461760000 trainer/global_step‚11449 % _timestamp‚1.7885095300578544e+09 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13695252089344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚933364910823312 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚978449857546240 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 239617686 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚933364893783952 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 239617684 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 143572954 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1078704514179072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 143572952 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1065009009367552 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13072493846141 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 239617686 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 _runtime‚13587.835176413 _step‚940 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13073401821145 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚965328305757696 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 290087003 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13695480871424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 290113347 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚69450814 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 290113347 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 213023768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚69450181 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚965328305757696 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760¬Ô«û4Ž./ _wandbruntime‚13588  _runtime‚13588sŸ€¸ .N OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1065099235072000 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13074519094631 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13073611119419 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 143585017 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13696451062272 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 213041504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚978533526080000 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _step‚941 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 290138204 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 213040893 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚965409942846464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚611 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚69456485 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 529749678 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 239637844 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13122673248768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 529749678 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1078795686134272 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 % _timestamp‚1.7885095310833752e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚946518447512487 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 529776050 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bye…p^tes.small_pool.current‚ 207797248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 290138204 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1065099259012096 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 239637846 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13123583233536 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 239637844Ÿ€'.•N L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 529776050 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 207797248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 290138204 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1065099259012096 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 239637846 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13123583233536 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 239637844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1065099235072000 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13074519094631 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13073611119419 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 143585017 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13696451062272 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 213041504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚978533526080000 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _step‚941 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 290138204 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 213040893 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚965409942846464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚611 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚69456485 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 529749678 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 239637844 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13122673248768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 529749678 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1078795686134272 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 % _timestamp‚1.7885095310833752e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚946518447512487 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0­Èùft4‘./ _wandbruntime‚13588  _runtime‚13588í§˜^’.Ø 1loss/SRqqt1/detection_loss‚2.4783261507749557 0loss/SRqqt4/detection_loss‚2.566756068728864 &loss/total_loss‚40.797684066585674 epoch‚9 trainer/global_step‚11449 0loss/FRt2/assignment_loss‚2.8346858894502796 1loss/SRqqt3/detection_loss‚0.8607487171916789 /loss/FBt2/detection_loss‚2.2549667003608884 % _timestamp‚1.7885095310856204e+09 /loss/FBt4/detection_loss‚2.3498859012668785 0loss/FRt3/assignment_loss‚1.3644740533828736 /loss/FRt4/assignment_loss‚3.213706051861798 1loss/FBt1/assignment_loss‚0.14621723936538664 /loss/FBt1/detection_loss‚0.6211092284992075 2loss/SRqqt3/assignment_loss‚0.8440343386685575 2loss/SRqqt4/assignment_loss‚1.5463321395218372 0loss/SRqqt2/detection_loss‚2.328197705745697 1loss/FBt3/assignment_loss‚0.21365817792684538 /loss/FRt2/detection_loss‚2.5005980771840424 .loss/FBt3/detection_loss‚2.593717187643051 _runtime‚13588.862674597 _step‚942 /loss/FRt3/detection_loss‚1.5251261864354213 /loss/FRt1/assignment_loss‚2.061545144570501 0loss/FBt2/assignment_loss‚0.1012852889026089 /loss/FBt4/assignment_loss‚0.395294050622066 /loss/FRt4/detection_loss‚2.9623809125688343 2loss/SRqqt1/assignment_loss‚1.1501998674124478 2loss/SRqqt2/assignment_loss‚1.8720148965716361 /loss/FRt1/detection_loss‚2.0124240919293173@óˆ c“.Ý _runtime‚13588.862674597 _step‚942 /loss/FRt3/detection_loss‚1.5251261864354213 /loss/FRt1/assignment_loss‚2.061545144570501 0loss/FBt2/assignment_loss‚0.1012852889026089 /loss/FBt4/assignment_loss‚0.395294050622066 /loss/FRt4/detection_loss‚2.9623809125688343 2loss/SRqqt1/assignment_loss‚1.1501998674124478 2loss/SRqqt2/assignment_loss‚1.8720148965716361 /loss/FRt1/detection_loss‚2.0124240919293173 1loss/SRqqt1/detection_loss‚2.4783261507749557 0loss/SRqqt4/detection_loss‚2.566756068728864 &loss/total_loss‚40.797684066585674 epoch‚9 trainer/global_step‚11449 0loss/FRt2/assignment_loss‚2.8346858894502796 1loss/SRqqt3/detection_loss‚0.8607487171916789 /loss/FBt2/detection_loss‚2.2549667003608884 % _timestamp‚1.7885095310856204e+09 /loss/FBt4/detection_loss‚2.3498859012668785 0loss/FRt3/assignment_loss‚1.3644740533828736 /loss/FRt4/assignment_loss‚3.213706051861798 1loss/FBt1/assignment_loss‚0.14621723936538664 /loss/FBt1/detection_loss‚0.6211092284992075 2loss/SRqqt3/assignment_loss‚0.8440343386685575 2loss/SRqqt4/assignment_loss‚1.5463321395218372 0loss/SRqqt2/detection_loss‚2.328197705745697 1loss/FBt3/assignment_loss‚0.21365817792684538 /loss/FRt2/detection_loss‚2.5005980771840424 .loss/FBt3/detection_loss‚2.593717187643051®î(ÎZ“”.‚(:ˆ ÜúéÔ♎l gpu.0.gpu‚25.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚122.153' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.53825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á»&"r•.‚(:ç ÜúéÔ£¼°l disk.dm-0.out‚187153.5859375 disk.nvme1n1.in‚ 744.57421875 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sdb1.in‚0 disk.sdb.out‚1516.9560546875 disk.nvme0n1.in‚ 939.6640625 disk.sda2.in‚ 9.19140625! disk.nvme1n1.out‚ 65362.828125 disk.sda.in‚ 9.19140625% proc.memory.availableMB‚ 1302.1875 cpu‚25.207547083464682 disk.dm-0.in‚ 1684.23828125 disk.sdb.in‚4.390625 disk.md0.in‚ 13.58203125 network.sent‚ 3.064739e+07 disk.md0.out‚ 1459.3828125 disk.sdb2.in‚4.390625 disk.sda1.out‚0 disk.rbd0.out‚ 3.87109375 disk.sdb1.out‚0# proc.memory.rssMB‚ 3012.59765625# disk.nvme0n1.out‚121790.7578125) disk./.usagePercent‚75.77737011648354! disk.sda2.out‚1516.9560546875 disk.rbd0.in‚0$ memory_percent‚92.05207824707031* proc.memory.percent‚18.387436866760254 proc.cpu.threads‚17 disk.md0p1.in‚ 13.58203125 disk.md0p1.out‚ 1459.3828125 disk.rbd2.out‚ 4.095703125 network.recv‚ 4.389509e+06$ disk./.usageGB‚2710.165802001953! disk.sdb2.out‚1516.9560546875 disk.sda.out‚1516.9560546875Ñóo–.‚(:ä ëúéÔâÚÏl disk.sdb1.in‚0 disk.md0.in‚ 13.58203125$ memory_percent‚92.04981327056885 cpu‚25.207372743799926% disk./.usageGB‚2709.6982917785645 disk.rbd2.out‚ 4.095703125 disk.rbd0.in‚0" disk.nvme0n1.out‚ 121853.015625 disk.nvme1n1.in‚ 744.57421875) proc.memory.availableMB‚ 1302.55859375! disk.sda.out‚1518.37255859375 disk.sda1.in‚0 disk.sdb.in‚4.390625* proc.memory.percent‚18.387436866760254" disk.sdb2.out‚1518.37255859375 disk.sdb1.out‚0 disk.md0p1.out‚1460.75" disk.sda2.out‚1518.37255859375! disk.nvme1n1.out‚ 65435.078125 disk.md0.out‚1460.75 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.dm-0.in‚ 1684.23828125 disk.dm-0.out‚ 187288.09375! disk.sdb.out‚1518.37255859375 network.recv‚ 4.394862e+06 disk.sda1.out‚0 disk.md0p1.in‚ 13.58203125 disk.sda2.in‚ 9.19140625# proc.memory.rssMB‚ 3012.59765625 network.sent‚ 3.0755009e+07 disk.sdb2.in‚4.390625 disk.nvme0n1.in‚ 939.6640625) disk./.usagePercent‚75.76429833496935 disk.sda.in‚ 9.19140625 disk.rbd0.out‚ 3.87109375ÎAÂ!“—.‚(:ˆ ëúéÔ¹ÛŸy gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚117.525' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.38125 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BD˜È’˜.‚(:‡ úúéÔüù—l gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚129.005' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.25125 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ2™ý|™.‚(:ñ úúéÔÏäˆm disk.sda2.in‚ 9.19140625 disk.nvme0n1.in‚ 939.6640625 disk.rbd2.in‚0 disk.sda.in‚ 9.19140625 disk.sda1.in‚0 disk.sdb.in‚4.390625( proc.memory.availableMB‚ 1302.4609375# proc.memory.rssMB‚ 3012.59765625 cpu‚25.20716140092142 disk.sda1.out‚0) disk./.usagePercent‚75.76955275276327# disk.nvme1n1.out‚65495.74609375 disk.sdb2.in‚4.390625! disk.sda.out‚1520.62060546875 disk.sdb1.out‚0 disk.md0p1.in‚ 13.58203125" disk.sda2.out‚1520.62060546875 disk.md0.in‚ 13.58203125 disk.md0.out‚ 1462.92578125 disk.rbd0.out‚ 3.87109375! disk.sdb.out‚1520.62060546875 disk.sdb1.in‚0! disk.dm-0.out‚187348.76171875# memory_percent‚92.0504093170166" disk.nvme0n1.out‚ 121853.015625 disk.nvme1n1.in‚ 744.57421875 disk.rbd2.out‚ 4.095703125" disk.sdb2.out‚1520.62060546875 disk.rbd0.in‚0$ disk./.usageGB‚2709.886215209961* proc.memory.percent‚18.387436866760254 proc.cpu.threads‚17 network.sent‚ 3.0761294e+07 network.recv‚ 4.399471e+06 disk.md0p1.out‚ 1462.92578125 disk.dm-0.in‚ 1684.23828125Úç«ëš.‚(:… ‰ûéÔØÇ÷l gpu.0.gpu‚10.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uøDjz›.‚(:ï ‰ûéÔääÊm# disk.nvme0n1.out‚121967.7734375 disk.md0.in‚ 13.58203125 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0p1.out‚ 1464.4609375# proc.memory.rssMB‚ 3012.59765625! disk.sda.out‚1522.22607421875! disk.nvme1n1.out‚ 65496.671875 disk.md0.out‚ 1464.4609375 network.recv‚ 4.403112e+06) disk./.usagePercent‚75.77480546398867 disk.nvme1n1.in‚ 744.57421875 disk.rbd2.in‚0 disk.rbd2.out‚ 4.095703125) proc.memory.availableMB‚ 1302.66796875 proc.cpu.threads‚17 network.sent‚ 3.0768779e+07 disk.sda1.in‚0 disk.md0p1.in‚ 13.58203125 disk.sda2.in‚ 9.19140625* proc.memory.percent‚18.387436866760254 disk.sdb2.in‚4.390625 disk.rbd0.out‚ 3.87109375 disk.dm-0.out‚187464.4453125 disk.sdb.in‚4.390625! disk.sdb.out‚1522.22607421875" disk.sda2.out‚1522.22607421875 cpu‚25.20696850710438$ disk./.usageGB‚2710.074077606201" disk.sdb2.out‚1522.22607421875 disk.sda.in‚ 9.19140625 disk.sdb1.in‚0 disk.dm-0.in‚ 1684.23828125$ memory_percent‚92.04914569854736 disk.nvme0n1.in‚ 939.6640625=dÉ34œ./ _wandbruntime‚13643  _runtime‚13643»™B,È. trainer/global_step‚11499 $ _timestamp‚1.788509586133023e+09 _runtime‚13643.910380971 _step‚943 & lr-AdamW/pg1‚7.641609314551603e-06 & lr-AdamW/pg2‚7.641609314551603e-06»káÍž.Ç trainer/global_step‚11499 $ _timestamp‚1.788509586133023e+09 _runtime‚13643.910380971 _step‚943 & lr-AdamW/pg1‚7.641609314551603e-06 & lr-AdamW/pg2‚7.641609314551603e-06¯Évû¶4Ÿ./ _wandbruntime‚13643  _runtime‚13643ÞûM| ( .…P ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 144176202 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 291358249 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚69740880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13128371101249 % _timestamp‚1.7885095861346366e+09 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 213917082 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13753762428928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚982587827236864 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚950444144612401 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13129279076253 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚969409278051840 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚937315773511152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 240625686 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚969409278051840 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _step‚944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚982586899262464 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 531957589 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13178549185024 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 291358249 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13177639213568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚937315790550512 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13177639213568 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 trainer/global_step‚11499 _runtime‚13643.91235781 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚646 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚950445069626765 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚982587827236864 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1083274057018880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1083274307643904 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚969409260048896 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 291331905 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13178549185024 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 240625684 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚982586899262464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13753989113856ì»$‘æ¡.ŠP WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚982586899262464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13753989113856 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 144176202 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 291358249 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚69740880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13128371101249 % _timestamp‚1.7885095861346366e+09 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 213917082 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13753762428928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚982587827236864 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚950444144612401 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13129279076253 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚969409278051840 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚937315773511152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 240625686 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚969409278051840 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _step‚944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚982586899262464 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 531957589 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13178549185024 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 291358249 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13177639213568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚937315790550512 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13177639213568 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 trainer/global_step‚11499 _runtime‚13643.91235781 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚646 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚950445069626765 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚982587827236864 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1083274057018880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1083274307643904 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚969409260048896 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 291331905 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13178549185024 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 240625684 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0°·fÜ$4¢./ _wandbruntime‚13645  _runtime‚13645’¥0Ý'£.N [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13754914259968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚937394808145056 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 532028952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 291383106 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚644 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚982669639808512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 213934923 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 532002580 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1069610520294400 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13755151417344 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚969490897137664 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 144188269 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚950524279480223 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 240645846 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 144188267 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚646 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1083365695651840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚969490879134720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚937394791105696 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 240645846 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 291356736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚982670567796224 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚982670567796224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69746656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13178760673792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚69747300 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13129488374527 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13179670658560 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 _step‚945 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 291356736 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚969490879134720 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13179670658560 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 % _timestamp‚1.7885095873141568e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 291383106 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268)ù5³'¤.•N OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 532002580 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1083365434554368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚969490897137664 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 144188269 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚950524279480223 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 240645846 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 144188267 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚646 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1083365695651840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚969490879134720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚937394791105696 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 240645846 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 291356736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚982670567796224 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚982670567796224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69746656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13178760673792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚69747300 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13129488374527 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13179670658560 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 _step‚945 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 291356736 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚969490879134720 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13179670658560 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 % _timestamp‚1.7885095873141568e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 291383106 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13754914259968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚937394808145056 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 532028952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 291383106 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚644 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚982669639808512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 213934923 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 532002580 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1069610520294400 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13755151417344 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 292603151 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 144779452 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚986723940978688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13812373262336 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚973490232343040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13233726638592 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 214809897 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚986723940978688 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _runtime‚13702.473452274 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13184248356357 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1074031579812352 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 trainer/global_step‚11549 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 292576807 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13234636610048K{*(µ.‰P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1074031579812352 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 trainer/global_step‚11549 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 292576807 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13234636610048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13185156331361 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚973490232343040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 241633684 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 534210491 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1087843953074688 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1074031603752448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚954451826609073 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚948 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 292603151 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 144779452 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚986723940978688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13812373262336 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚973490232343040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13233726638592 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 214809897 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚986723940978688 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _runtime‚13702.473452274 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13184248356357´–Ó4¶./ _wandbruntime‚13703  _runtime‚13703EÃòw·.N EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚973571833425920 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 144791517 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚627 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 214828383 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1074121805516800 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 292601638 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70036864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70036239 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13235758083584 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13813540001792 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13186273604847 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 trainer/global_step‚11549 _runtime‚13703.417975921 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚986807609512448 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 $ _timestamp‚1.788509645640041e+09 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 214827756 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13813768770560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 534281854 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1087935598227456 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚954531961477103 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13185365629635 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 144791519 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 241653846 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13234848098816 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 241653844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 534281854Ü@p'¸.”N WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 292601638 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70036864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70036239 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13235758083584 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13813540001792 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13186273604847 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 trainer/global_step‚11549 _runtime‚13703.417975921 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚986807609512448 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 $ _timestamp‚1.788509645640041e+09 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 214827756 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13813768770560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 534281854 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1087935598227456 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚954531961477103 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13185365629635 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 144791519 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 241653846 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13234848098816 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 241653844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 534281854 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚973571833425920 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 144791517 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚627 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 214828383 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1074121805516800µq€zî4¹./ _wandbruntime‚13703  _runtime‚13703‘¸ºìVº.Ð .loss/FRt2/detection_loss‚2.097687486266973 .loss/FRt3/detection_loss‚1.551969196110104 1loss/SRqqt3/detection_loss‚0.8938540200858066 /loss/FBt2/detection_loss‚2.1882143389611017 .loss/FBt3/detection_loss‚2.107452920892022 _runtime‚ 13703.4199602 1loss/SRqqt1/assignment_loss‚1.259370611758814 1loss/FBt1/assignment_loss‚0.20794170751785368 0loss/FBt2/assignment_loss‚0.2885292704929184 0loss/SRqqt2/detection_loss‚2.365040240287781 /loss/FBt1/detection_loss‚0.6956432918973622 epoch‚9 % _timestamp‚1.7885096456429663e+09 0loss/FRt2/assignment_loss‚2.9648777012731515 2loss/SRqqt3/assignment_loss‚1.1314298497745767 /loss/FRt1/assignment_loss‚1.915748646725779 0loss/FBt4/assignment_loss‚0.1973675605145485 0loss/FRt4/assignment_loss‚3.4967147988431595 /loss/FRt1/detection_loss‚1.7036450408706276 1loss/SRqqt1/detection_loss‚2.0050279029778073 1loss/SRqqt4/detection_loss‚2.2236180161607675 trainer/global_step‚11549 .loss/FRt4/detection_loss‚2.760809532859746 /loss/FRt3/assignment_loss‚1.771920702778376 2loss/SRqqt2/assignment_loss‚0.8980927827954293 0loss/SRqqt4/assignment_loss‚1.24797872453928 /loss/FBt4/detection_loss‚2.3713894346181084 _step‚950 %loss/total_loss‚38.48668007068337 0loss/FBt3/assignment_loss‚0.1423562916812774£Þ‚[».Õ .loss/FRt4/detection_loss‚2.760809532859746 /loss/FRt3/assignment_loss‚1.771920702778376 2loss/SRqqt2/assignment_loss‚0.8980927827954293 0loss/SRqqt4/assignment_loss‚1.24797872453928 /loss/FBt4/detection_loss‚2.3713894346181084 _step‚950 %loss/total_loss‚38.48668007068337 0loss/FBt3/assignment_loss‚0.1423562916812774 .loss/FRt2/detection_loss‚2.097687486266973 .loss/FRt3/detection_loss‚1.551969196110104 1loss/SRqqt3/detection_loss‚0.8938540200858066 /loss/FBt2/detection_loss‚2.1882143389611017 .loss/FBt3/detection_loss‚2.107452920892022 _runtime‚ 13703.4199602 1loss/SRqqt1/assignment_loss‚1.259370611758814 1loss/FBt1/assignment_loss‚0.20794170751785368 0loss/FBt2/assignment_loss‚0.2885292704929184 0loss/SRqqt2/detection_loss‚2.365040240287781 /loss/FBt1/detection_loss‚0.6956432918973622 epoch‚9 % _timestamp‚1.7885096456429663e+09 0loss/FRt2/assignment_loss‚2.9648777012731515 2loss/SRqqt3/assignment_loss‚1.1314298497745767 /loss/FRt1/assignment_loss‚1.915748646725779 0loss/FBt4/assignment_loss‚0.1973675605145485 0loss/FRt4/assignment_loss‚3.4967147988431595 /loss/FRt1/detection_loss‚1.7036450408706276 1loss/SRqqt1/detection_loss‚2.0050279029778073 1loss/SRqqt4/detection_loss‚2.2236180161607675 trainer/global_step‚11549¶©ˆTÍ’¼.‚(:‡ ÔûéÔ›»ªn gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.434' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.8585 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n—`~½.‚(:ó ÔûéÔíŠân disk.nvme1n1.in‚ 744.57421875 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.dm-0.in‚ 1684.23828125! disk.dm-0.out‚188335.88671875! disk.sdb.out‚1530.52099609375" disk.sdb2.out‚1530.52099609375 disk.sda.in‚ 9.19140625 disk.sda1.in‚0 disk.sda2.in‚ 9.19140625 disk.md0.out‚ 1472.42578125$ memory_percent‚92.05307960510254( proc.memory.availableMB‚ 1302.0234375 network.sent‚ 3.0897846e+07 network.recv‚ 4.425854e+06 disk.rbd2.out‚ 4.095703125 disk.nvme0n1.in‚ 939.6640625 cpu‚25.205997655662276 disk.sdb2.in‚4.390625 disk.sda1.out‚0 disk.md0p1.in‚ 13.58203125 disk.md0p1.out‚ 1472.42578125 proc.cpu.threads‚17$ disk./.usageGB‚2710.915657043457! disk.sda.out‚1530.52099609375 disk.sdb.in‚4.390625# disk.nvme1n1.out‚65498.21484375 disk.md0.in‚ 13.58203125 disk.sdb1.out‚0* proc.memory.percent‚18.387436866760254) disk./.usagePercent‚75.79833637728271 disk.rbd0.out‚ 3.87109375" disk.sda2.out‚1530.52099609375# proc.memory.rssMB‚ 3012.59765625" disk.nvme0n1.out‚ 122837.671875PØYª“¾.‚(:ˆ ãûéÔÏïÄm gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚120.927' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.23175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Óô…‚t¿.‚(:é ãûéÔº€—o disk.sda2.in‚ 9.19140625 disk.rbd2.in‚0 disk.md0p1.out‚ 1474.46875 disk.dm-0.out‚ 188654.28125 proc.cpu.threads‚17( disk./.usagePercent‚75.7983370172459# disk.nvme1n1.out‚65498.80078125 disk.sdb1.in‚0) proc.memory.availableMB‚ 1302.01953125 network.recv‚ 4.431352e+06 disk.nvme0n1.in‚ 939.6640625 disk.md0.out‚ 1474.46875 disk.rbd2.out‚ 4.095703125 network.sent‚ 3.1007872e+07! disk.sda2.out‚1532.6298828125 disk.sda.in‚ 9.19140625 disk.sda.out‚1532.6298828125 disk.sdb.in‚4.390625 disk.sda1.in‚0 disk.rbd0.out‚ 3.87109375 disk.dm-0.in‚ 1684.23828125 cpu‚25.205807733632568% disk./.usageGB‚2710.9156799316406 disk.sdb1.out‚0 disk.md0p1.in‚ 13.58203125$ disk.nvme0n1.out‚123160.23046875 disk.nvme1n1.in‚ 744.57421875 disk.sdb2.in‚4.390625! disk.sdb2.out‚1532.6298828125 disk.sda1.out‚0 disk.sdb.out‚1532.6298828125$ memory_percent‚92.05310344696045 disk.md0.in‚ 13.58203125 disk.rbd0.in‚0# proc.memory.rssMB‚ 3012.59765625* proc.memory.percent‚18.387436866760254{ÕÓ’À.‚(:‡ òûéÔµ •m gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.395' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.09875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º’¥8~Á.‚(:ó òûéÔð‹óm network.sent‚ 3.1013285e+07$ disk./.usageGB‚2711.166679382324! disk.sda.out‚1534.02294921875# proc.memory.rssMB‚ 3012.59765625" disk.sdb2.out‚1534.02294921875 disk.sda1.in‚0 disk.md0p1.out‚ 1475.80078125! disk.nvme1n1.out‚ 65498.984375 disk.md0.in‚ 13.58203125 disk.rbd2.in‚0* proc.memory.percent‚18.387436866760254 disk.rbd0.in‚0 disk.nvme1n1.in‚ 744.57421875) disk./.usagePercent‚75.80535506693845" disk.sda2.out‚1534.02294921875$ disk.nvme0n1.out‚123272.41796875 disk.sda.in‚ 9.19140625 disk.rbd0.out‚ 3.87109375 disk.md0p1.in‚ 13.58203125 disk.sdb.in‚4.390625 disk.nvme0n1.in‚ 939.6640625 disk.rbd2.out‚ 4.095703125 disk.dm-0.in‚ 1684.23828125! disk.dm-0.out‚188771.40234375 disk.sda2.in‚ 9.19140625 disk.md0.out‚ 1475.80078125 disk.sdb2.in‚4.390625 disk.sdb1.in‚0 disk.sdb1.out‚0$ memory_percent‚92.05071926116943 network.recv‚ 4.435237e+06! disk.sdb.out‚1534.02294921875 disk.sda1.out‚0) proc.memory.availableMB‚ 1302.41015625 cpu‚25.20560651343589 proc.cpu.threads‚17í½Î‘Â.‚(:† üéÔ¹ãâl gpu.0.gpu‚58.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚175.512' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-Æ;ÁwÃ.‚(:ì üéÔùŒ˜m( disk./.usagePercent‚75.7757545294035 disk.sdb1.out‚0 disk.md0p1.out‚ 1477.30078125 disk.sda2.in‚ 9.19140625 disk.rbd2.in‚0 disk.rbd0.out‚ 3.87109375 proc.cpu.threads‚17! disk.sdb.out‚1535.58251953125 disk.md0.out‚ 1477.30078125 disk.rbd0.in‚0# proc.memory.rssMB‚ 3012.59765625 disk.md0p1.in‚ 13.58203125 disk.dm-0.in‚ 1684.23828125 disk.dm-0.out‚ 188996.859375 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚18.387436866760254 network.sent‚ 3.1018562e+07% disk./.usageGB‚2710.1080207824707 disk.sdb1.in‚0 disk.nvme1n1.in‚ 744.57421875 disk.md0.in‚ 13.58203125$ memory_percent‚92.05009937286377 disk.sda.in‚ 9.19140625! disk.sda.out‚1535.58251953125) proc.memory.availableMB‚ 1302.51171875" disk.sda2.out‚1535.58251953125 disk.nvme0n1.out‚ 123497.75 cpu‚25.205421724501008 disk.sdb.in‚4.390625 disk.sdb2.in‚4.390625" disk.sdb2.out‚1535.58251953125 network.recv‚ 4.438499e+06 disk.nvme0n1.in‚ 939.6640625! disk.nvme1n1.out‚ 65499.109375 disk.rbd2.out‚ 4.095703125ˆfË4Ä./ _wandbruntime‚13758  _runtime‚13758ÝŒ®åÉÅ.à & lr-AdamW/pg1‚5.278364998853772e-06 & lr-AdamW/pg2‚5.278364998853772e-06 trainer/global_step‚11599 % _timestamp‚1.7885097002981415e+09 _runtime‚13758.075371842 _step‚951.…,åÎÆ.È % _timestamp‚1.7885097002981415e+09 _runtime‚13758.075371842 _step‚951 & lr-AdamW/pg1‚5.278364998853772e-06 & lr-AdamW/pg2‚5.278364998853772e-06 trainer/global_step‚11599·|t!N4Ç./ _wandbruntime‚13758  _runtime‚13758&#?·$È.†P KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚945217532965552 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1092414170213376 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 242641686 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1078542865034752 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 293821709 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚977571168631296 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 145382702 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚11599 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13290724035072 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚958458583591381 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚977571186634240 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 293848053 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚649 _step‚952 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13871069233664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13240125611465 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 215703739 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13871281238528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚958457658577017 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚990861910669312 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚990860982694912 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13289814063616 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚990860982694912 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 536489739 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1092413934268416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 536463393 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13290724035072 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 242641684 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 215704390 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235944960 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D>DeviceStatsMonitor.on³ÚU_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 242641686 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚651 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13241033586469 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 293821709 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0ãf(É.‹P M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 242641684 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚977571186634240 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1078542888974848 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚977571168631296 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13240125611465 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 215703739 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13871281238528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚958457658577017 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚990861910669312 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚990860982694912 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13289814063616 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚990860982694912 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 536489739 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1092413934268416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 536463393 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13290724035072 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 242641684 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 215704390 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235944960 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 242641686 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚651 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13241033586469 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 293821709 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚945217532965552 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1092414170213376 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 242641686 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1078542865034752 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 293821709 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚977571168631296 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 145382702 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚11599 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13290724035072 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚958458583591381 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚977571186634240 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 293848053 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚649 _step‚952 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13871069233664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 536534756 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70327546 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚977652805720064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13291845508608 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚977652787717120 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885097012325547e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚990943723240960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13290935523840 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚977652805720064 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 242661844 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13290935523840 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚958537793444839 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚640 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1078633090739200 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13872466145792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244320256 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13241242884743 trainer/global_step‚11599 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13291845508608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 215721673 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 242661846 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13872245765632 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚945296550560096õ§H¾'Ì.•N ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚945296550560096 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚642 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 145394767 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 536508384 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 215722315 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1092505336504832 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚990944651228672 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 536534756 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70327546 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚977652805720064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13291845508608 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚977652787717120 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885097012325547e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚990943723240960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13290935523840 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚977652805720064 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 242661844 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13290935523840 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚958537793444839 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚640 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1078633090739200 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13872466145792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244320256 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13241242884743 trainer/global_step‚11599 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13291845508608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 215721673 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 242661846 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13872245765632 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944¹Î¥²4Í./ _wandbruntime‚13759  _runtime‚13759|*œJ`Î.Ú 2loss/SRqqt1/assignment_loss‚1.2740286301954515 1loss/FBt4/assignment_loss‚0.20738563696721712 0loss/SRqqt4/detection_loss‚2.822608468325242 trainer/global_step‚11599 2loss/SRqqt2/assignment_loss‚1.3872219901159406 2loss/SRqqt4/assignment_loss‚1.4372233702274768 /loss/FBt3/detection_loss‚2.2485402897000313 .loss/FBt4/detection_loss‚2.053188044577837 _step‚954 &loss/total_loss‚38.509386929822455 0loss/FRt1/assignment_loss‚1.7465220050113957 0loss/FRt3/assignment_loss‚1.2494974074692562 0loss/SRqqt1/detection_loss‚2.072067674781595 _runtime‚13759.012403003 2loss/SRqqt3/assignment_loss‚0.9250121290431075 /loss/FRt3/detection_loss‚1.8057877251814152 % _timestamp‚1.7885097012353964e+09 1loss/FBt3/assignment_loss‚0.04853166267093911 0loss/FRt2/assignment_loss‚2.8232605820116787 0loss/FRt4/assignment_loss‚2.8169163543245066 1loss/FBt1/assignment_loss‚0.12845443767672624 /loss/FRt2/detection_loss‚2.1721857736980463 /loss/FRt4/detection_loss‚3.0923864685970806 1loss/FBt2/assignment_loss‚0.25528590608652924 0loss/SRqqt2/detection_loss‚2.438025906085968 .loss/FBt1/detection_loss‚0.682824077454638 .loss/FBt2/detection_loss‚2.142135469118754 /loss/FRt1/detection_loss‚1.6914421122708563 1loss/SRqqt3/detection_loss‚0.9888548082307629 epoch‚9U˜š"iÏ.ß trainer/global_step‚11599 2loss/SRqqt2/assignment_loss‚1.3872219901159406 2loss/SRqqt4/ø2Ê\üassignment_loss‚1.4372233702274768 /loss/FBt3/detection_loss‚2.2485402897000313 .loss/FBt4/detection_loss‚2.053188044577837 _step‚954 &loss/total_loss‚38.509386929822455 0loss/FRt1/assignment_loss‚1.7465220050113957 0loss/FRt3/assignment_loss‚1.2494974074692562 0loss/SRqqt1/detection_loss‚2.072067674781595 _runtime‚13759.012403003 2loss/SRqqt3/assignment_loss‚0.9250121290431075 /loss/FRt3/detection_loss‚1.8057877251814152 % _timestamp‚1.7885097012353964e+09 1loss/FBt3/assignment_loss‚0.04853166267093911 0loss/FRt2/assignment_loss‚2.8232605820116787 0loss/FRt4/assignment_loss‚2.8169163543245066 1loss/FBt1/assignment_loss‚0.12845443767672624 /loss/FRt2/detection_loss‚2.1721857736980463 /loss/FRt4/detection_loss‚3.0923864685970806 1loss/FBt2/assignment_loss‚0.25528590608652924 0loss/SRqqt2/detection_loss‚2.438025906085968 .loss/FBt1/detection_loss‚0.682824077454638 .loss/FBt2/detection_loss‚2.142135469118754 /loss/FRt1/detection_loss‚1.6914421122708563 1loss/SRqqt3/detection_loss‚0.9888548082307629 epoch‚9 2loss/SRqqt1/assignment_loss‚1.2740286301954515 1loss/FBt4/assignment_loss‚0.20738563696721712 0loss/SRqqt4/detection_loss‚2.822608468325242º*éÉ“Ð.‚(:ˆ üéÔÂœ³m gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚170.995' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.74875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û®Œ1Ñ.‚(:ô üéÔª¼÷n disk.rbd0.out‚ 3.87109375 disk.md0p1.out‚ 1478.84765625 disk.dm-0.in‚ 1684.23828125# disk.nvme1n1.out‚65499.36328125 disk.md0.in‚ 13.58203125 disk.sda2.in‚ 9.19140625) proc.memory.availableMB‚ 1302.55078125 proc.cpu.threads‚17 disk.sda.in‚ 9.19140625 disk.md0p1.in‚ 13.58203125 disk.sdb.in‚4.390625 disk.rbd2.in‚0$ disk./.usageGB‚2710.107494354248! disk.sda.out‚1537.18701171875 disk.sda1.out‚0 disk.sdb1.in‚0 disk.sdb1.out‚0* proc.memory.percent‚18.387436866760254 cpu‚25.205209810235868 network.sent‚ 3.1035199e+07 disk.md0.out‚ 1478.84765625$ memory_percent‚92.04986095428467) disk./.usagePercent‚75.77573981025006 disk.sda1.in‚0" disk.nvme0n1.out‚ 123702.046875# proc.memory.rssMB‚ 3012.59765625 disk.rbd2.out‚ 4.095703125 network.recv‚ 4.442984e+06! disk.dm-0.out‚189201.41015625! disk.sdb.out‚1537.18701171875 disk.nvme0n1.in‚ 939.6640625" disk.sdb2.out‚1537.18701171875 disk.rbd0.in‚0" disk.sda2.out‚1537.18701171875 disk.nvme1n1.in‚ 744.57421875 disk.sdb2.in‚4.390625;Wqm‘Ò.‚(:† ŸüéÔÍšðm gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.09' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.7725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eðÒ€Ó.‚(:õ ŸüéÔìá°n disk.sda2.in‚ 9.19140625" disk.sda2.out‚1539.07763671875# disk.nvme0n1.out‚123810.0546875 disk.md0.out‚ 1480.67578125 disk.sdb2.in‚4.390625) disk./.usagePercent‚75.77574183680017 disk.sda1.in‚0$ memory_percent‚92.04990863800049 disk.md0.in‚ 13.58203125 disk.sda.in‚ 9.19140625 disk.sdb.in‚4.390625# proc.memory.rssMB‚ 3012.59765625 disk.nvme1n1.in‚ 744.57421875 disk.rbd2.out‚ 4.095703125! disk.sda.out‚1539.07763671875 disk.md0p1.in‚ 13.58203125 proc.cpu.threads‚17" disk.sdb2.out‚1539.07763671875 disk.rbd0.in‚0 disk.sdb1.in‚0$ disk./.usageGB‚2710.107566833496 disk.nvme0n1.in‚ 939.76171875! disk.sdb.out‚1539.07763671875 disk.md0p1.out‚ 1480.67578125 disk.dm-0.in‚ 1684.3359375! disk.dm-0.out‚189309.63671875* proc.memory.percent‚18.387436866760254 network.sent‚ 3.1129962e+07 disk.sda1.out‚0) proc.memory.availableMB‚ 1302.54296875 network.recv‚ 4.448287e+06# disk.nvme1n1.out‚65499.58203125 disk.rbd0.out‚ 3.87109375 disk.sdb1.out‚0 cpu‚25.205025243737147 disk.rbd2.in‚0…pvΑÔ.‚(:† ®üéÔ‡÷Ðk gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚39.0 gpu.0.powerWatts‚175.844' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.961 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨ çþsÕ.‚(:è ®üéÔ¿ÈÀl disk.dm-0.in‚ 1684.3359375 disk.sdb.in‚4.390625" disk.nvme0n1.out‚ 123811.140625 disk.sda2.in‚ 9.19140625 disk.rbd2.out‚ 4.095703125 disk.md0.in‚ 13.58203125 disk.rbd0.in‚0$ memory_percent‚92.04988479614258 disk.sdb2.in‚4.390625" disk.nvme1n1.out‚ 65499.8203125 disk.rbd2.in‚0% disk./.usageGB‚2710.0085639953613' proc.memory.availableMB‚ 1302.546875# proc.memory.rssMB‚ 3012.59765625 proc.cpu.threads‚17 disk.sdb2.out‚1541.037109375 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 939.76171875 disk.md0.out‚ 1482.55859375 network.sent‚ 3.1136261e+07 network.recv‚ 4.452942e+06) disk./.usagePercent‚75.77297367601001 disk.sda.out‚1541.037109375 disk.sdb1.out‚0 disk.md0p1.in‚ 13.58203125 disk.dm-0.out‚189310.9609375 disk.sdb.out‚1541.037109375 disk.sda.in‚ 9.19140625 disk.sdb1.in‚0 disk.md0p1.out‚ 1482.55859375 disk.sda2.out‚1541.037109375 disk.nvme1n1.in‚ 744.57421875* proc.memory.percent‚18.387436866760254 cpu‚25.20480995107664 disk.rbd0.out‚ 3.87109375áÍSÿ4Ö./ _wandbruntime‚13810  _runtime‚13810^ž´£Ë×.Å % _timestamp‚1.7885097525958924e+09 _runtime‚13810.373106363 _step‚955 ' lr-AdamW/pg1‚4.2594570610961675e-06 ' lr-AdamW/pg2‚4.2594570610961675e-06 trainer/global_step‚11649…qÐØ.Ê ' lr-AdamW/pg1‚4.2594570610961675e-06 ' lr-AdamW/pg2‚4.2594570610961675e-06 trainer/global_step‚11649 % _timestamp‚1.7885097525958924e+09 _runtime‚13810.373106363 _step‚955»ž:4Ù./ _wandbruntime‚13810  _runtime‚138104‚ÔŽ (Ú.†P QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 243649684 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 216599696 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1096983814672896 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13929664415744 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚646 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚13810.374157527 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 295066611 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚949168429732112 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242236416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚994998024411136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 538742641 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 295092955 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚981652140925440 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13345901488640 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 243649686 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1083054150257152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13296910841577 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218296320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 295066611 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13345901488640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚648 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚994998024411136 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7885097525968783e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 145985952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13929882712064 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 145985954 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 538716295 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚994998952385536 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _step‚956 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚949168412692752 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚981652140925440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚981652122922496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 243649684 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1083054174197248 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 243649686 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70613742 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚962465340573689 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 216599048 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1096984056909312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚11649 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚981652122922496G|ãQ(Û.‹P QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 295066611 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚949168429732112 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242236416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚994998024411136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 538742641 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 295092955 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚981652140925440 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13345901488640 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 243649686 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1083054150257152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13296910841577 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218296320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 295066611 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13345901488640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚648 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚994998024411136 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7885097525968783e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 145985952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13929882712064 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 145985954 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 538716295 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚994998952385536 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _step‚956 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚949168412692752 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚981652140925440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚981652122922496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 243649684 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1083054174197248 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 243649686 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70613742 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚962465340573689 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 216599048 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1096984056909312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚11649 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚981652122922496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 243649684 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 216599696 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1096983814672896 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13929664415744 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚646 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚13810.374157527 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249¼SÀ¿4Ü./ _wandbruntime‚13811  _runtime‚13811)žã Ý.N WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚995081692944896 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 243669846 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 243669844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚981733742008320 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚981733742008320 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 538787658 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1083144375961600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 295091442 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚655 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚962545475441719 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13297120139851 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚995081692944896 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 145998017 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚995080764957184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13930838610944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚981733760011264 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1083144399901696 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13298028115063 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _step‚957 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 trainer/global_step‚11649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70619517 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13931061088256 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 538787658 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70618864 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.al'e)²öl.freed‚1097075214572544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚949247430287296 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 295117812 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13347932933632 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 295117812 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚653 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1097075460989952 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 538761286 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 243669846 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13347022948864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 243669844 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚13811.453167952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚981733760011264 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13347022948864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 216617536 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 216616881 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚133479329336320ÅÃr'Þ.•N S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚962544550427147 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 295091442 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚981733760011264 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1083144399901696 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13298028115063 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _step‚957 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 trainer/global_step‚11649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70619517 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13931061088256 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 538787658 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70618864 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1097075214572544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚949247430287296 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 295117812 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13347932933632 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 295117812 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚653 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1097075460989952 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 538761286 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 243669846 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13347022948864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 243669844 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚13811.453167952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚981733760011264 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13347022948864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 216617536 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 216616881 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13347932933632 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚995081692944896 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 243669846 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 243669844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚981733742008320 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚981733742008320 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 538787658 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1083144375961600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 295091442 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚655 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚962545475441719 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13297120139851 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚995081692944896 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 145998017 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚995080764957184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13930838610944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 % _timestamp‚1.7885098110721734e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚985733077213696 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 244657686 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚665 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 244657686 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚999135066127360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚966472097555997 _step‚960 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 540995543 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1101553744968192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13351880121681 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚985733095216640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚953119309459312 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13402898885120 WGDeviceStatsMonitor.on_train_batch_staªÚ¥Ârt/allocated_bytes.large_pool.peak‚ 13002869760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393472 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 244657684 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚985733077213696 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚966471172541633 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1087565435479552 trainer/global_step‚11699 _runtime‚13868.849663101rÿGÙ(ï.‹P HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 217493400 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13988309488640 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13352788096685 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13402898885120 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 % _timestamp‚1.7885098110721734e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚985733077213696 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 244657686 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚665 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 244657686 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚999135066127360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚966472097555997 _step‚960 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 540995543 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1101553744968192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13351880121681 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚985733095216640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚953119309459312 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13402898885120 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393472 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 244657684 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚985733077213696 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚966471172541633 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1087565435479552 trainer/global_step‚11699 _runtime‚13868.849663101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚999135994101760 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 296311513ÀuwÂì4ð./ _wandbruntime‚13869  _runtime‚13869õÉÖÜ'ñ.N I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 217511932 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13352997394959 % _timestamp‚1.7885098120208285e+09 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 146601269 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1101645389831168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 244677844 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚985814696299520 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚999218734661120 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚999217806673408 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 296336344 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚11699 _runtime‚13869.799229863 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 541040560 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚985814714302464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚650 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70910663 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1087655685124096 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13404020358656 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 244677846 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 217511282 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 541014188 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 296362714 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13989478035456 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 244677844 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚985814696299520 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚648 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1087655661184000 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13989704707072 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13353905370171 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 244677846 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 296336344 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1101645139219456 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 296362714 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚966551307409455 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚985814714302464 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚966552232424027 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚953198310014496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 541040560 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70910015 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 146601267 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚999218734661120 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _step‚961 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0‰i°Ô'ò.•N QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70910015 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 146601267 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚999218734661120 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _step‚961 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 217511932 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13352997394959 % _timestamp‚1.7885098120208285e+09 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 146601269 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1101645389831168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 244677844 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚985814696299520 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚999218734661120 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚999217806673408 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 296336344 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚11699 _runtime‚13869.799229863 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 541040560 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚985814714302464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚650 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70910663 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1087655685124096 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13404020358656 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 244677846 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 217511282 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 541014188 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 296362714 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13989478035456 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 244677844 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚985814696299520 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚648 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1087655661184000 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13989704707072 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13353905370171 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 244677846 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 296336344 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1101645139219456 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 296362714 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚966551307409455 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚985814714302464 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚966552232424027 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚953198310014496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 541040560Áeði4ó./ _wandbruntime‚13869  _runtime‚13869½í1cô.Ø 2loss/SRqqt3/assignment_loss‚0.9570149152122182 3loss/FBt3/assignment_loss‚0.0028487386589404196 /loss/FBt3/detection_loss‚2.9289228320121765 epoch‚9 _runtime‚13869.80078169 0loss/FBt1/assignment_loss‚0.1578699085596409 1loss/FBt4/assignment_loss‚0.03277880947867247 /loss/FRt2/detection_loss‚2.3375936900063055 0loss/SRqqt2/detection_loss‚2.394509425530067 0loss/FRt1/assignment_loss‚2.2764231581240892 0loss/FRt4/assignment_loss‚2.8832992116610208 2loss/SRqqt2/assignment_loss‚1.5842801904162536 2loss/SRqqt4/assignment_loss‚1.4340535299409003 0loss/FBt2/assignment_loss‚0.2492286071234907 /loss/FRt3/detection_loss‚1.6193740331567823 1loss/SRqqt1/detection_loss‚1.7047478599207742 /loss/FBt1/detection_loss‚0.6521004880901033 1loss/SRqqt4/detection_loss‚2.4964262332235063 %loss/toEGÝ-ûtal_loss‚41.39992646591325 /loss/FRt3/assignment_loss‚1.342040721575419 1loss/SRqqt3/detection_loss‚1.0140695570534546 .loss/FBt2/detection_loss‚2.652159465683831 /loss/FBt4/detection_loss‚2.9029975425113332 _step‚962 % _timestamp‚1.7885098120237634e+09 trainer/global_step‚11699 -loss/FRt1/detection_loss‚2.07928395831253 0loss/FRt2/assignment_loss‚2.9522437699463056 2loss/SRqqt1/assignment_loss‚1.4362133159967405 .loss/FRt4/detection_loss‚3.309446503718694·)ø}cõ.Ý /loss/FRt3/assignment_loss‚1.342040721575419 1loss/SRqqt3/detection_loss‚1.0140695570534546 .loss/FBt2/detection_loss‚2.652159465683831 /loss/FBt4/detection_loss‚2.9029975425113332 _step‚962 % _timestamp‚1.7885098120237634e+09 trainer/global_step‚11699 -loss/FRt1/detection_loss‚2.07928395831253 0loss/FRt2/assignment_loss‚2.9522437699463056 2loss/SRqqt1/assignment_loss‚1.4362133159967405 .loss/FRt4/detection_loss‚3.309446503718694 2loss/SRqqt3/assignment_loss‚0.9570149152122182 3loss/FBt3/assignment_loss‚0.0028487386589404196 /loss/FBt3/detection_loss‚2.9289228320121765 epoch‚9 _runtime‚13869.80078169 0loss/FBt1/assignment_loss‚0.1578699085596409 1loss/FBt4/assignment_loss‚0.03277880947867247 /loss/FRt2/detection_loss‚2.3375936900063055 0loss/SRqqt2/detection_loss‚2.394509425530067 0loss/FRt1/assignment_loss‚2.2764231581240892 0loss/FRt4/assignment_loss‚2.8832992116610208 2loss/SRqqt2/assignment_loss‚1.5842801904162536 2loss/SRqqt4/assignment_loss‚1.4340535299409003 0loss/FBt2/assignment_loss‚0.2492286071234907 /loss/FRt3/detection_loss‚1.6193740331567823 1loss/SRqqt1/detection_loss‚1.7047478599207742 /loss/FBt1/detection_loss‚0.6521004880901033 1loss/SRqqt4/detection_loss‚2.4964262332235063 %loss/total_loss‚41.39992646591325Âáþ&‡’ö.‚(:‡ ùüéÔ綼n gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚158.746' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.6865 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©nytq÷.‚(:æ ùüéÔš³˜o disk.rbd2.out‚ 4.095703125) proc.memory.availableMB‚ 1302.25390625 disk.sda.out‚ 1549.80859375 disk.rbd0.in‚0# proc.memory.rssMB‚ 3012.59765625! disk.nvme1n1.out‚ 65504.609375 disk.md0.in‚ 13.58203125 disk.rbd2.in‚0 disk.sda.in‚ 9.19140625 disk.sda1.in‚0 disk.sda2.out‚ 1549.80859375 proc.cpu.threads‚17) disk./.usagePercent‚75.77728948112126$ disk.nvme0n1.out‚124478.13671875 disk.nvme1n1.in‚ 744.57421875 disk.sdb2.out‚ 1549.80859375$ memory_percent‚92.05167293548584 cpu‚25.20387454921915 disk.md0p1.out‚ 1491.0234375 network.sent‚ 3.1266731e+07 network.recv‚ 4.474276e+06 disk.sdb1.out‚0% disk./.usageGB‚2710.1629180908203 disk.sdb1.in‚0 disk.sdb.in‚4.390625* proc.memory.percent‚18.387436866760254 disk.nvme0n1.in‚ 939.76171875 disk.md0.out‚ 1491.0234375 disk.rbd0.out‚ 3.87109375 disk.md0p1.in‚ 13.58203125 disk.dm-0.in‚ 1684.3359375! disk.dm-0.out‚189982.74609375 disk.sdb.out‚ 1549.80859375 disk.sdb2.in‚4.390625 disk.sda1.out‚0 disk.sda2.in‚ 9.19140625lÝRÚ‘ø.‚(:† ˆýéÔè‘ùl gpu.0.gpu‚32.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚135.184' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a4Ië{ù.‚(:ð ˆýéÔµïîm" disk.sdb2.out‚1551.31689453125 disk.rbd0.in‚0 proc.cpu.threads‚17 network.sent‚ 3.1376639e+07 disk.sda1.out‚0 disk.sdb1.in‚0 disk.md0p1.out‚ 1492.47265625 disk.dm-0.in‚ 1684.3359375 disk.sdb.in‚4.390625 disk.dm-0.out‚ 190111.1875 network.recv‚ 4.480092e+06! disk.sda.out‚1551.31689453125 disk.rbd0.out‚ 3.87109375! disk.sdb.out‚1551.31689453125 disk.nvme1n1.in‚ 744.57421875 disk.md0.in‚ 13.58203125 disk.md0.out‚ 1492.47265625$ memory_percent‚92.05164909362793* proc.memory.percent‚18.387436866760254" disk.sda2.out‚1551.31689453125# disk.nvme1n1.out‚65504.80078125 disk.rbd2.out‚ 4.095703125# proc.memory.rssMB‚ 3012.59765625$ disk./.usageGB‚2710.667812347412 disk.rbd2.in‚0 disk.sda.in‚ 9.19140625 disk.sda1.in‚0( proc.memory.availableMB‚ 1302.2578125 disk.nvme0n1.in‚ 939.76171875$ disk.nvme0n1.out‚124606.38671875 disk.sdb2.in‚4.390625 disk.sdb1.out‚0 disk.md0p1.in‚ 13.58203125 cpu‚25.20367776048435) disk./.usagePercent‚75.79140653585023 disk.sda2.in‚ 9.19140625-Þ,X’ú.‚(:‡ —ýéÔ±µÁm gpu.0.gpu‚5.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚142.653' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.66325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z>uû.‚(:ê —ýéÔË…n$ disk./.usageGB‚2710.792537689209 disk.sda2.in‚ 9.19140625 disk.sda2.out‚1552.759765625 disk.nvme1n1.in‚ 744.57421875 disk.rbd2.in‚0 disk.rbd2.out‚ 4.095703125* proc.memory.percent‚18.387436866760254 network.recv‚ 4.484137e+06 disk.sda.out‚1552.759765625 disk.sda1.out‚0 disk.dm-0.in‚ 1684.3359375 disk.md0.in‚ 13.58203125) disk./.usagePercent‚75.79489390860847 disk.sdb2.in‚4.390625 disk.sda1.in‚0 disk.rbd0.in‚0! disk.dm-0.out‚190283.68359375# disk.nvme1n1.out‚65505.12109375$ memory_percent‚92.05048084259033 proc.cpu.threads‚17 disk.sdb.out‚1552.759765625) proc.memory.availableMB‚ 1302.44921875 network.sent‚ 3.1382694e+07 disk.md0p1.out‚ 1493.84765625 disk.nvme0n1.out‚ 124778.5625# proc.memory.rssMB‚ 3012.59765625 disk.sda.in‚ 9.19140625 disk.sdb.in‚4.390625 disk.md0.out‚ 1493.84765625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sdb2.out‚1552.759765625 disk.rbd0.out‚ 3.87109375 disk.md0p1.in‚ 13.58203125 disk.nvme0n1.in‚ 939.76171875 cpu‚25.203475745914044n=š™“ü.‚(:ˆ ¦ýéÔëô™m gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚159.419' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.85475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hú«ký.‚(:à ¦ýéÔþ‘ým% disk./.usageGB‚2710.9155197143555 disk.sda2.out‚1554.232421875 disk.sdb1.out‚0# proc.memory.rssMB‚ 3012.59765625 proc.cpu.threads‚17$ disk.nvme0n1.out‚125033.04296875 disk.md0.in‚ 13.58203125 disk.sdb2.out‚1554.232421875 disk.sda.out‚1554.232421875( proc.memory.availableMB‚ 1302.1953125 cpu‚25.20329296250373 disk.sdb2.in‚4.390625 disk.sdb.in‚4.390625* proc.memory.percent‚18.387436866760254 network.sent‚ 3.1386811e+07 disk.md0.out‚1495.25) disk./.usagePercent‚75.79833253750355 disk.nvme1n1.in‚ 744.57421875 disk.rbd2.in‚0 disk.md0p1.in‚ 13.58203125 disk.md0p1.out‚1495.25 disk.dm-0.in‚ 1684.3359375! disk.dm-0.out‚190538.34765625 network.recv‚ 4.488747e+06 disk.nvme0n1.in‚ 939.76171875 disk.rbd0.in‚0 disk.sdb1.in‚0" disk.nvme1n1.out‚ 65505.3046875 disk.sda.in‚ 9.19140625 disk.sda1.in‚0 disk.rbd0.out‚ 3.87109375 disk.sdb.out‚1554.232421875$ memory_percent‚92.05203056335449 disk.sda2.in‚ 9.19140625 disk.rbd2.out‚ 4.095703125 disk.sda1.out‚0Q žL4þ./ _wandbruntime‚13927  _runtime‚13927úò3Èÿ. & lr-AdamW/pg1‚2.548131275947638e-06 & lr-AdamW/pg2‚2.548131275947638e-06 trainer/global_step‚11749 % _timestamp‚1.7885098694788568e+09 _runtime‚13927.25615244 _step‚963qÞ…ÈÍ€/Ç & lr-AdamW/pg2‚2.548131275947638e-06 trainer/global_step‚11749 % _timestamp‚1.7885098694788568e+09 _runtime‚13927.25615244 _step‚963 & lr-AdamW/pg1‚2.548131275947638e-06æo ]4// _wandbruntime‚13927  _runtime‚13927^V(‚/ŠP \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13458986310144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71195903 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1003272107843584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13407757376789 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚644 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13408665351793 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _step‚964 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 147192452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 543222099 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13458076338688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚989814049507840 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14046915898368 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 218388357 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 218387713 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚989814031504896 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 245665686 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 245665686 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1003273035817984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚642 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14047138388992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚970478854538305 trainer/global_step‚11749 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 297556415 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1106123883031040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 297556415 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚970477929523941 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1106123636600320 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚989814031504896 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚957070189186512 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13458076338688 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13458986310144 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 245665684 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 297582759 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71195261 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚957070172147152 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 147192454 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 245665684 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 543248445 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1003273035817984 _runtime‚13927.2577063562MHµ(ƒ/P JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚644 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13408665351793 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _step‚964 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 147192452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 543222099 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13458076338688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚989814049507840 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14046915898368 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 218388357 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 218387713 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚989814031504896 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 245665686 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 245665686 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1003273035817984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚642 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14047138388992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚970478854538305 trainer/global_step‚11749 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 297556415 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1106123883031040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 297556415 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚970477929523941 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1106123636600320 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚989814031504896 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚957070189186512 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13458076338688 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13458986310144 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 245665684 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 297582759 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71195261 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚957070172147152 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 147192454 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 245665684 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 543248445 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1003273035817984 _runtime‚13927.257706356 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13458986310144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71195903 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1003272107843584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13407757376789 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0ÄḊÕ4„// _wandbruntime‚13928  _runtime‚13928RµPIº …/”N BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStat‘;ý`sMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1003355776377344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚970558989406335 trainer/global_step‚11749 _step‚965 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚989895650590720 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 543293462 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 543267090 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71201711 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13459197798912 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 218405587 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 % _timestamp‚1.7885098704361832e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1092166970346496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 543293462 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 297607616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚71201070 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14048311003136 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 245685846 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13409782625279 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚970558064391763 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚989895650590720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1106215030737920 _runtime‚13928.213886348 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 245685844 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚970558064391763 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚989895650590720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250611712 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1106215030737920 _runtime‚13928.213886348 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 245685844 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1003355776377344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚970558989406335 trainer/global_step‚11749 _step‚965 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚989895650590720 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 543293462 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 543267090 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71201711 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13459197798912 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 218405587 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 % _timestamp‚1.7885098704361832e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1092166970346496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 543293462 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 297607616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚71201070 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14048311003136 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 245685846 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13409782625279 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185Å°ÞµP4‡// _wandbruntime‚13928  _runtime‚13928P=xYˆ/Ó 1loss/FBt4/assignment_loss‚0.24736925087159356 trainer/global_step‚11749 0loss/FRt2/assignment_loss‚2.6174232378238584 2loss/SRqqt3/assignment_loss‚0.9746175654828548 1loss/SRqqt1/detection_loss‚1.6999804617897156 0loss/SRqqt2/detection_loss‚2.365135832266374 /loss/FBt4/detection_loss‚2.0476584169599743 /loss/FRt3/assignment_loss‚1.346121370792389 2loss/SRqqt1/assignment_loss‚1.4771557081070157 1loss/FBt1/assignment_loss‚0.16638610394021142 0loss/FBt2/assignment_loss‚0.3040622321158348 _step‚966 0loss/FRt4/assignment_loss‚2.9005529842069073 1loss/SRqqt3/detection_loss‚0.8327627777457237 -loss/FBt2/detection_loss‚2.58161469300588 % _timestamp‚1.7885098704389222e+09 /loss/FRt1/assignment_loss‚1.576980403491429 2loss/SRqqt2/assignment_loss‚1.4026061752528856 .loss/FRt4/detection_loss‚2.626624411152255 2loss/SRqqt4/assignment_loss‚1.4258004773873836 /loss/FBt3/assignment_loss‚0.211643938072181 .loss/FRt3/detection_loss‚1.381321850973062 1loss/SRqqt4/detection_loss‚2.3870241791009903 /loss/FBt1/detection_loss‚0.5710365399252624 .loss/FBt3/detection_loss‚1.982216113143497 &loss/total_loss‚37.339437642696794 /loss/FRt2/detection_loss‚2.3344577567242997 -loss/FRt1/detection_loss‚1.87888516236522 epoch‚9 _runtime‚13928.215910067ëth^‰/Ø trainer/global_step‚11749 0loss/FRt2/assignment_loss‚2.6174232378238584 2loss/SRqqt3/assignment_loss‚0.9746175654828548 1loss/SRqqt1/detection_loss‚1.6999804617897156 0loss/SRqqt2/detection_loss‚2.365135832266374 /loss/FBt4/detection_loss‚2.0476584169599743 /loss/FRt3/assignment_loss‚1.346121370792389 2loss/SRqqt1/assignment_loss‚1.4771557081070157 1loss/FBt1/assignment_loss‚0.16638610394021142 0loss/FBt2/assignment_loss‚0.3040622321158348 _step‚966 0loss/FRt4/assignment_loss‚2.9005529842069073 1loss/SRqqt3/detection_loss‚0.8327627777457237 -loss/FBt2/detection_loss‚2.58161469300588 % _timestamp‚1.7885098704389222e+09 /loss/FRt1/assignment_loss‚1.576980403491429 2loss/SRqqt2/assignment_loss‚1.4026061752528856 .loss/FRt4/detection_loss‚2.626624411152255 2loss/SRqqt4/assignment_loss‚1.4258004773873836 /loss/FBt3/assignment_loss‚0.211643938072181 .loss/FRt3/detection_loss‚1.381321850973062 1loss/SRqqt4/detection_loss‚2.3870241791009903 /loss/FBt1/detection_loss‚0.5710365399252624 .loss/FBt3/detection_loss‚1.982216113143497 &loss/total_loss‚37.339437642696794 /loss/FRt2/detection_loss‚2.3344577567242997 -loss/FRt1/detection_loss‚1.87888516236522 epoch‚9 _runtime‚13928.215910067 1loss/FBt4/assignment_loss‚0.24736925087159356ÆeòxŠ/‚(:… µýéԉÑm gpu.0.gpu‚28.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚146.38' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.595 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tƒV}‹/‚(:ò µýéÔðž¨n! disk.sda.out‚1556.53173828125 disk.md0p1.out‚ 1497.4765625) disk./.usagePercent‚75.80536519968899 disk.md0.out‚ 1497.4765625 disk.dm-0.in‚ 1684.3359375 disk.sdb.in‚4.390625 network.recv‚ 4.492137e+06 disk.sdb2.in‚4.390625 disk.sdb1.in‚0$ memory_percent‚92.05214977264404% disk./.usageGB‚2711.1670417785645 disk.nvme1n1.in‚ 744.57421875# disk.nvme1n1.out‚65505.57421875 disk.md0p1.in‚ 13.58203125! disk.dm-0.out‚190650.48046875 disk.sda2.in‚ 9.19140625! disk.nvme0n1.out‚ 125144.90625 disk.rbd2.out‚ 4.095703125 disk.rbd0.in‚0 disk.nvme0n1.in‚ 939.76171875) proc.memory.availableMB‚ 1302.17578125# proc.memory.rssMB‚ 3012.59765625 network.sent‚ 3.1392277e+07 disk.md0.in‚ 13.58203125 disk.rbd2.in‚0! disk.sdb.out‚1556.53173828125" disk.sda2.out‚1556.53173828125* proc.memory.percent‚18.387436866760254" disk.sdb2.out‚1556.53173828125 disk.sda.in‚ 9.19140625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚ 3.87109375 cpu‚25.203109058885406 disk.sdb1.out‚0 proc.cpu.threads‚17ßêZš“Œ/‚(:ˆ ÄýéÔ†‹Æm gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.825' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.70625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#1d÷q/‚(:æ ÄýéÔƒàn disk.md0p1.out‚ 1499.15625 disk.dm-0.out‚ 191009.171875 disk.sda2.out‚1558.283203125# disk.nvme1n1.out‚65505.76171875# proc.memory.rssMB‚ 3012.59765625 disk.rbd0.out‚ 3.87109375 disk.sdb.out‚1558.283203125 disk.rbd2.out‚ 4.095703125 disk.sdb2.out‚1558.283203125( proc.memory.availableMB‚ 1302.1796875% disk./.usageGB‚2711.1659660339355 disk.sdb1.out‚0 disk.md0.in‚ 13.58203125 cpu‚25.202889440401943 disk.nvme1n1.in‚ 744.57421875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.md0p1.in‚ 13.58203125 disk.dm-0.in‚ 1684.3359375 disk.sdb.in‚4.390625$ disk.nvme0n1.out‚125503.41015625 disk.sdb2.in‚4.390625 disk.sda2.in‚ 9.19140625 disk.md0.out‚ 1499.15625 disk.rbd2.in‚0 disk.sda.in‚ 9.19140625 disk.sda.out‚1558.283203125$ memory_percent‚92.05212593078613* proc.memory.percent‚18.387436866760254 proc.cpu.threads‚17) disk./.usagePercent‚75.80533512141893 disk.nvme0n1.in‚ 939.76171875 network.sent‚ 3.1499987e+07 network.recv‚ 4.498166e+06 disk.sda1.in‚0sw«ÆpŽ/‚(:å ÓýéÔˆ¤ m) disk./.usagePercent‚75.80533896119809 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 1684.3359375# disk.nvme1n1.out‚65506.12109375 disk.sda.in‚ 9.19140625$ memory_percent‚92.05281734466553 disk.nvme1n1.in‚ 744.57421875 disk.dm-0.out‚191009.7109375# proc.memory.rssMB‚ 3012.59765625 network.recv‚ 4.50232e+06 disk.sda.out‚1559.8076171875 disk.md0p1.out‚1500.625) proc.memory.availableMB‚ 1302.06640625 cpu‚25.20269541931083$ disk./.usageGB‚2711.166103363037$ disk.nvme0n1.out‚125503.58984375 disk.rbd2.out‚ 4.095703125 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sdb2.in‚4.390625 disk.md0p1.in‚ 13.58203125 disk.sdb.out‚1559.8076171875 network.sent‚ 3.1508059e+07 disk.nvme0n1.in‚ 939.76171875 disk.md0.out‚1500.625 disk.rbd2.in‚0 disk.sda1.in‚0 disk.rbd0.out‚ 3.87109375 disk.sdb.in‚4.390625 disk.sda2.in‚ 9.19140625! disk.sda2.out‚1559.8076171875 disk.md0.in‚ 13.58203125! disk.sdb2.out‚1559.8076171875 disk.rbd0.in‚0* proc.memory.percent‚18.387436866760254Ί“/‚(:ˆ ÓýéÔ“²Èm gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.715' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.17875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Çi¸Ù/‚(:… âýéÔä«Äk gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.22' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.055 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"zÚr‘/‚(:ç âýéÔ¨ÔÛl cpu‚25.202516201220565 proc.cpu.threads‚17 disk.md0.out‚ 1502.59765625 disk.rbd2.in‚0 disk.sda2.in‚ 9.19140625 disk.md0.in‚ 13.58203125 disk.sda1.in‚0 network.recv‚ 4.507027e+06) disk./.usagePercent‚75.80533842789544 disk.sdb.in‚4.390625 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1684.3359375 disk.sdb.out‚1561.845703125 disk.rbd0.out‚ 3.87109375 disk.md0p1.in‚ 13.58203125* proc.memory.percent‚18.387436866760254 disk.sda.in‚ 9.19140625 disk.sda.out‚1561.845703125! disk.dm-0.out‚191476.37890625$ memory_percent‚92.05436706542969% proc.memory.availableMB‚ 1301.8125$ disk.nvme0n1.out‚125970.03515625 disk.sdb1.in‚0 disk.md0p1.out‚ 1502.59765625 network.sent‚ 3.1514344e+07 disk.nvme0n1.in‚ 939.76171875 disk.nvme1n1.in‚ 744.57421875 disk.rbd2.out‚ 4.095703125# proc.memory.rssMB‚ 3012.59765625$ disk./.usageGB‚2711.166084289551 disk.sda2.out‚1561.845703125 disk.nvme1n1.out‚ 65506.34375 disk.sdb2.in‚4.390625 disk.sdb2.out‚1561.845703125 disk.sdb1.out‚0ú¨'B4’// _wandbruntime‚13984  _runtime‚13984o…B Ë“/Å ' lr-AdamW/pg1‚1.8560700662854246e-06 ' lr-AdamW/pg2‚1.8560700662854246e-06 trainer/global_step‚11799 % _timestamp‚1.7885099268298488e+09 _runtime‚13984.607071234 _step‚967! YöД/Ê % _timestamp‚1.7885099268298488e+09 _runtime‚13984.607071234 _step‚967 ' lr-AdamW/pg1‚1.8560700662854246e-06 ' lr-AdamW/pg2‚1.8560700662854246e-06 trainer/global_step‚11799ÇÀx Y4•// _wandbruntime‚13984  _runtime‚13984NÈ=ºÿ–/ŠP D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 219283524 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚961021068913712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1110693695721984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 545501347 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13463634631897 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚993895003799040 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13514163763712 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1007409149559808 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14105689797632 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚974484686506249 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 545475001 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1007409149559808 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235944960 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 246673684 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚993894985796096 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 147795704 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1007410077534208 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13514163763712 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 147795702 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13515073735168 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13515073735168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1007410077534208 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peakƒ&? ‚ 184953856 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _runtime‚13984.608193309 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 298827661 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚630 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 298801317 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1096588005924352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1007409149559808 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14105689797632 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚974484686506249 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 545475001 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1007409149559808 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235944960 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 246673684 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚993894985796096 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 147795704 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1007410077534208 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13514163763712 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 147795702 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13515073735168 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13515073735168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1007410077534208 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _runtime‚13984.608193309 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 298827661 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚630 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 298801317 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1096588005924352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 219301376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚71493609 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14106836687872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚652 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1096678231628800 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚961100069468896 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚654 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1096678255568896 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 298826148 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13516195208704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1110785333608448 _step‚969 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 545546364 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 147807769 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11799 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 219302030 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚974564821374071 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 298826148 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚993976622884864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 265291776 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚961100086508256 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 246693846 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚993976604881920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 298852518 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚993976604881920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 246693844 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13515285223936 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1007492818093568 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14107078039552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚974565746388643 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 219301376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚71493609 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14106836687872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚652 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1096678231628800 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚961100069468896 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚654 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1096678255568896 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 298826148 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13516195208704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1110785333608448 _step‚969 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 545546364 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 147807769 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11799 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 219302030 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚974564821374071 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 298826148 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚993976622884864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 265291776 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚961100086508256 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 246693846 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚993976604881920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 298852518 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.sm›— ´all_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚993976604881920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 246693844 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13515285223936 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1007492818093568 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14107078039552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚974565746388643ɼk†‡4›// _wandbruntime‚13985  _runtime‚139850h'÷aœ/Û 0loss/FRt4/assignment_loss‚3.1885815048217774 1loss/FBt2/assignment_loss‚0.38917313363057815 %loss/total_loss‚41.21625852715444 1loss/FBt4/assignment_loss‚0.10888726752420133 1loss/SRqqt3/detection_loss‚0.9372393895454448 /loss/SRqqt4/detection_loss‚2.70357448640077 .loss/FBt2/detection_loss‚2.189459686477979 epoch‚9 1loss/SRqqt1/assignment_loss‚1.323116584122181 /loss/FRt4/detection_loss‚2.9980915212631225 1loss/SRqqt2/detection_loss‚2.8648301793469324 0loss/FBt1/assignment_loss‚0.1421360880350961 /loss/FRt2/detection_loss‚2.5318646280627166 % _timestamp‚1.7885099277895992e+09 3loss/FBt3/assignment_loss‚0.0018459987418282253 2loss/SRqqt4/assignment_loss‚1.0844576616326105 _step‚970 /loss/FRt3/detection_loss‚1.6671790353587417 /loss/FBt4/detection_loss‚2.4456249246230493 _runtime‚13985.566738571 1loss/SRqqt2/assignment_loss‚1.606409004992909 2loss/SRqqt3/assignment_loss‚0.9504043147738638 1loss/SRqqt1/detection_loss‚2.3818584784865378 /loss/FBt1/detection_loss‚0.7105897903888369 /loss/FBt3/detection_loss‚2.4549286365509033 0loss/FRt2/assignment_loss‚2.9095229918048497 /loss/FRt1/detection_loss‚1.6762705412348138 trainer/global_step‚11799 0loss/FRt1/assignment_loss‚2.3687166419140127 0loss/FRt3/assignment_loss‚1.5814960374206792X«¼öf/à 1loss/FBt4/assignment_loss‚0.10888726752420133 1loss/SRqqt3/detection_loss‚0.9372393895454448 /loss/SRqqt4/detection_loss‚2.70357448640077 .loss/FBt2/detection_loss‚2.189459686477979 epoch‚9 1loss/SRqqt1/assignment_loss‚1.323116584122181 /loss/FRt4/detection_loss‚2.9980915212631225 1loss/SRqqt2/detection_loss‚2.8648301793469324 0loss/FBt1/assignment_loss‚0.1421360880350961 /loss/FRt2/detection_loss‚2.5318646280627166 % _timestamp‚1.7885099277895992e+09 3loss/FBt3/assignment_loss‚0.0018459987418282253 2loss/SRqqt4/assignment_loss‚1.0844576616326105 _step‚970 /loss/FRt3/detection_loss‚1.6671790353587417 /loss/FBt4/detection_loss‚2.4456249246230493 _runtime‚13985.566738571 1loss/SRqqt2/assignment_loss‚1.606409004992909 2loss/SRqqt3/assignment_loss‚0.9504043147738638 1loss/SRqqt1/detection_loss‚2.3818584784865378 /loss/FBt1/detection_loss‚0.7105897903888369 /loss/FBt3/detection_loss‚2.4549286365509033 0loss/FRt2/assignment_loss‚2.9095229918048497 /loss/FRt1/detection_loss‚1.6762705412348138 trainer/global_step‚11799 0loss/FRt1/assignment_loss‚2.3687166419140127 0loss/FRt3/assignment_loss‚1.5814960374206792 0loss/FRt4/assignment_loss‚3.1885815048217774 1loss/FBt2/assignment_loss‚0.38917313363057815 %loss/total_loss‚41.21625852715444Ê6NTö€ž/‚(:õ ñýéÔèšÞm" disk.nvme1n1.out‚ 65506.8984375 network.recv‚ 4.510514e+06$ disk./.usageGB‚2710.107982635498 disk.sdb2.in‚4.390625 disk.rbd0.out‚ 3.87109375 disk.sdb1.out‚0 disk.nvme0n1.in‚ 939.76171875 proc.cpu.threads‚17) disk./.usagePercent‚75.77575346279816 disk.sda.in‚ 9.19140625 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.rbd2.in‚0! disk.sda.out‚1563.18505859375! disk.dm-0.out‚191637.66015625! disk.sdb.out‚1563.18505859375 disk.sda2.in‚ 9.19140625$ disk.nvme0n1.out‚126130.76171875 disk.md0.in‚ 13.58203125# proc.memory.rssMB‚ 3012.59765625* proc.memory.percent‚18.387436866760254 network.sent‚ 3.1530343e+07 disk.sdb.in‚4.390625$ memory_percent‚92.05410480499268 disk.md0p1.in‚ 13.58203125 disk.md0p1.out‚ 1503.87890625 disk.dm-0.in‚ 1684.3359375 disk.nvme1n1.in‚ 744.57421875 disk.sda1.in‚0" disk.sda2.out‚1563.18505859375 disk.md0.out‚ 1503.87890625) proc.memory.availableMB‚ 1301.85546875 disk.rbd2.out‚ 4.095703125" disk.sdb2.out‚1563.18505859375 disk.sda1.out‚0 cpu‚25.202295545397835·Ôß|žŸ/‚(:“ ñýéÔô›š gpu.0.gpu‚23.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚119.978' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.994499999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y´uEy /‚(:î €þéÔº±÷l disk.sda.in‚ 9.19140625 disk.sdb1.out‚0 network.recv‚ 4.516182e+06 disk.sda.out‚1565.107421875 disk.md0.in‚ 13.58203125# proc.memory.rssMB‚ 3012.59765625 proc.cpu.threads‚17 disk.sda2.out‚1565.107421875 disk.nvme1n1.in‚ 744.57421875 network.sent‚ 3.1625216e+07 disk.rbd2.in‚0 disk.sdb2.out‚1565.107421875! disk.dm-0.out‚191681.66796875 disk.sdb.in‚4.390625 disk.md0.out‚ 1505.73046875 disk.rbd2.out‚ 4.095703125 disk.dm-0.in‚ 1684.3359375 disk.sdb.out‚1565.107421875" disk.nvme1n1.out‚ 65507.1015625) proc.memory.availableMB‚ 1301.85546875 disk.sdb2.in‚4.390625 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 939.76171875$ memory_percent‚92.05410480499268) disk./.usagePercent‚75.76093788824434 disk.sdb1.in‚0 cpu‚25.202100168110114 disk.rbd0.out‚ 3.87109375 disk.md0p1.in‚ 13.58203125 disk.md0p1.out‚ 1505.73046875 disk.rbd0.in‚0 disk.sda2.in‚ 9.19140625$ disk.nvme0n1.out‚126174.56640625* proc.memory.percent‚18.387436866760254% disk./.usageGB‚2709.5781059265137Óã%“¡/‚(:ˆ €þéÔûˆ™m gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚117.193' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.29825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú‹a’¢/‚(:‡ þéÔ‘óåk gpu.0.gpu‚34.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚156.578' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.1445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z®Tv£/‚(:ë þéÔþÑül disk.nvme1n1.in‚ 744.66015625 disk.sdb2.out‚1566.841796875 disk.sda1.out‚0' proc.memory.availableMB‚ 1301.859375 network.sent‚ 3.1630138e+07 disk.md0.out‚ 1507.38671875 disk.sdb.out‚1566.841796875* proc.memory.percent‚18.387436866760254 proc.cpu.threads‚17 disk.sda2.in‚ 9.19140625$ disk.nvme0n1.out‚126175.69921875 disk.sdb1.out‚0 disk.md0p1.out‚ 1507.38671875! disk.dm-0.out‚191683.56640625 disk.sdb.in‚4.390625$ memory_percent‚92.05408096313477% disk./.usageGB‚2709.5779609680176 disk.sdb2.in‚4.390625# proc.memory.rssMB‚ 3012.59765625) disk./.usagePercent‚75.76093383514412 disk.sda2.out‚1566.841796875 disk.rbd0.in‚0 disk.dm-0.in‚ 1685.171875 network.recv‚ 4.519552e+06 disk.md0.in‚ 13.58203125 disk.sda.in‚ 9.19140625 disk.sdb1.in‚0 cpu‚25.201902077150745 disk.rbd2.out‚ 4.095703125 disk.sda1.in‚0 disk.rbd0.out‚ 3.87109375 disk.md0p1.in‚ 13.58203125 disk.nvme0n1.in‚ 940.51171875" disk.nvme1n1.out‚ 65507.8671875 disk.rbd2.in‚0 disk.sda.out‚1566.8417968757&ÿü4¤// _wandbruntime‚14036  _runtime‚14036O %3Ë¥/Å _step‚971 ' lr-AdamW/pg1‚1.2732765249770012e-06 ' lr-AdamW/pg2‚1.2732765249770012e-06 trainer/global_step‚11849 % _timestamp‚1.7885099790953875e+09 _runtime‚14036.872615645™L$Ц/Ê % _timestamp‚1.7885099790953875e+09 _runtime‚14036.872615645 _step‚971 ' lr-AdamW/pg1‚1.2732765249770012e-06 ' lr-AdamW/pg2‚1.2732765249770012e-06 trainer/global_step‚11849ËÃ4Íy4§// _wandbruntime‚14036  _runtime‚14036#Mè´(¨/‰P TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71779639 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚978492368502921 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 _step‚972 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242236416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚964971948640912 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 _runtime‚14036.87371836 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 247681686 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13571161160192 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 300046219 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13570251188736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 247681686 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1115263681252352 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14164390105600 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 220179248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚655 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 247681684 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71780294 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚11849 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚997975940087296 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1101099291146752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1011547119250432 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 547754249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13520419862009 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 547727903 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13519511887005 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1011546191276032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚997975958090240 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1011546191276032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚657 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚978491443488557 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 247681686 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13571161160192 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 300046219 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13570251188736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 247681686 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1115263681252352 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14164390105600 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 220179248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚655 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 247681684 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71780294 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚11849 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚997975940087296 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1101099291146752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1011547119250432 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 547754249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13520419862009 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 547727903 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13519511887005 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1011546191276032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚997975958090240 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1011546191276032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚657 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0ÌK5u"4ª// _wandbruntime‚14037  _runtime‚14037Šæ)sG «/”N T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1011628931822080 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 300097420 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚998057559173120 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚965050949196096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1101189540791296 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 _runtime‚14037.828001473 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 247701846 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚998057577176064 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 247701846 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238028800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13520629160283 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13571372648960 ICDeviceStatsMonitor.onÍX5oÓ_train_batch_end/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 300071050 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 547799266 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214088704 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1011629859809792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚71785427 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚973 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 547772894 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 300071050 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1115355334231552 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚978572503370951 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚978571578356379 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14165579351552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13571372648960 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1011629859809792 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 247701844 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71786065 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚965050966235456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 220196444 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1101189516851200 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚638 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13521537135495 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13572282633728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚998057577176064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14165793440256 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 trainer/global_step‚11849 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 547799266 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 148411017 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 247701844 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13572282633728 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 300097420 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7885099800500348e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚998057559173120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚640 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 148411019 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712Sœ8Ð'¬/™N OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 247701846 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚998057577176064 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 247701846 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238028800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13520629160283 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13571372648960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 300071050 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 547799266 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214088704 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1011629859809792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚71785427 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚973 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 547772894 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 300071050 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1115355334231552 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚978572503370951 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚978571578356379 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14165579351552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13571372648960 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1011629859809792 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 247701844 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71786065 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚965050966235456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 220196444 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1101189516851200 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚638 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13521537135495 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13572282633728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚998057577176064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14165793440256 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 trainer/global_step‚11849 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 547799266 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 148411017 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 247701844 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13572282633728 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 300097420 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7885099800500348e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚998057559173120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚640 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 148411019 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1011628931822080 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 300097420 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚998057559173120 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚965050949196096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1101189540791296 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 _runtime‚14037.828001473Íqå^94­// _wandbruntime‚14037  _runtime‚14037Ô]¤äX®/Ò 2loss/SRqqt3/assignment_loss‚0.8839233599120009 1loss/FBt2/assignment_loss‚0.30498987783266784 /loss/FRt3/detection_loss‚1.6606045546190393 -loss/FBt2/detection_loss‚2.32261061668396 0loss/FRt2/assignment_loss‚3.0772260253628096 0loss/FRt3/assignment_loss‚1.3796451043698095 1loss/SRqqt4/assignment_loss‚1.161379048279647 /loss/SRqqt1/detection_loss‚2.31517155603929 1loss/SRqqt2/detection_loss‚2.5349630202565874 /loss/FBt1/detection_loss‚0.7478387069798285 epoch‚9 1loss/FBt4/assignment_loss‚0.14598786810130251 .loss/FRt2/detection_loss‚2.091470268322155 &loss/total_loss‚39.692738165305514 trainer/global_step‚11849 0loss/FRt4/assignment_loss‚3.1683919429779053 2loss/SRqqt2/assignment_loss‚1.3667012435339747 1loss/FBt1/assignment_loss‚0.20121749413369394 /loss/FBt3/detection_loss‚1.6571665455897648 % _timestamp‚1.7885099800526335e+09 .loss/FRt1/detection_loss‚1.903095164897386 1loss/SRqqt3/detection_loss‚0.8779425025126315 0loss/SRqqt4/detection_loss‚2.321306290297673 1loss/SRqqt1/assignment_loss‚1.391934827308763 0loss/FBt3/assignment_loss‚0.1151109850877674 /loss/FRt4/detection_loss‚3.5152343809604645 .loss/FBt4/detection_loss‚2.211748037073347 _runtime‚14037.82964933 _step‚974 .loss/FRt1/assignment_loss‚2.33707874417305Øü^Ô]¯/× 0loss/FRt4/assignment_loss‚3.1683919429779053 2loss/SRqqt2/assignment_loss‚1.3667012435339747 1loss/FBt1/assignment_loss‚0.20121749413369394 /loss/FBt3/detection_loss‚1.6571665455897648 % _timestamp‚1.7885099800526335e+09 .loss/FRt1/detection_loss‚1.903095164897386 1loss/SRqqt3/detection_loss‚0.8779425025126315 0loss/SRqqt4/detection_loss‚2.321306290297673 1loss/SRqqt1/assignment_loss‚1.391934827308763 0loss/FBt3/assignment_loss‚0.1151109850877674 /loss/FRt4/detection_loss‚3.5152343809604645 .loss/FBt4/detection_loss‚2.211748037073347 _runtime‚14037.82964933 _step‚974 .loss/FRt1/assignment_loss‚2.33707874417305 2loss/SRqqt3/assignment_loss‚0.8839233599120009 1loss/FBt2/assignment_loss‚0.30498987783266784 /loss/FRt3/detection_loss‚1.6606045546190393 -loss/FBt2/detection_loss‚2.32261061668396 0loss/FRt2/assignment_loss‚3.0772260253628096 0loss/FRt3/assignment_loss‚1.3796451043698095 1loss/SRqqt4/assignment_loss‚1.161379048279647 /loss/SRqqt1/detection_loss‚2.31517155603929 1loss/SRqqt2/detection_loss‚2.5349630202565874 /loss/FBt1/detection_loss‚0.7478387069798285 epoch‚9 1loss/FBt4/assignment_loss‚0.14598786810130251 .loss/FRt2/detection_loss‚2.091470268322155 &loss/total_loss‚39.692738165305514 trainer/global_step‚11849ÎóU;ôj°/‚(:ß žþéÔü†Ïl) disk./.usagePercent‚75.75641302849013$ disk./.usageGB‚2709.416275024414 disk.sda2.out‚ 1568.73828125# disk.nvme1n1.out‚65508.85546875 disk.md0.in‚ 13.58203125 disk.sdb2.in‚4.390625 disk.sda.out‚ 1568.73828125 disk.sda1.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚ 13.58203125 disk.sdb.out‚ 1568.73828125 proc.cpu.threads‚17 disk.rbd2.out‚ 4.095703125 network.recv‚ 4.52427e+06 disk.md0.out‚ 1509.2109375 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1685.171875) proc.memory.availableMB‚ 1301.88671875 disk.nvme1n1.in‚ 744.66015625 disk.sda.in‚ 9.19140625 disk.sdb.in‚4.390625 disk.nvme0n1.out‚ 126236.75! disk.dm-0.out‚191745.60546875 cpu‚25.201705845833917 network.sent‚ 3.1636432e+07 disk.sda2.in‚ 9.19140625 disk.rbd0.out‚ 3.87109375# memory_percent‚92.0539140701294* proc.memory.percent‚18.387436866760254 disk.sdb2.out‚ 1568.73828125 disk.sda1.out‚0 disk.sdb1.in‚0# proc.memory.rssMB‚ 3012.59765625 disk.nvme0n1.in‚ 940.51171875 disk.md0p1.out‚ 1509.2109375Ȧ“±/‚(:ˆ žþéÔÖ×õl gpu.0.gpu‚12.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚179.579' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.89475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KéÇx‘²/‚(:† ­þéÔ› ×k gpu.0.gpu‚29.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.028' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.007 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 <|ç}³/‚(:ò ­þéÔÕêÞl cpu‚25.20150829352299! disk.sda.out‚1570.28076171875 disk.md0.out‚ 1510.69921875 disk.rbd2.in‚0 disk.sdb2.in‚4.390625 disk.sdb.in‚4.390625" disk.sda2.out‚1570.28076171875 disk.nvme0n1.in‚ 940.51171875# proc.memory.rssMB‚ 3012.59765625 network.sent‚ 3.1743454e+07 disk.sda1.in‚0 disk.sdb1.in‚0 disk.md0p1.in‚ 13.58203125 disk.nvme1n1.in‚ 744.66015625) proc.memory.availableMB‚ 1301.81640625$ disk./.usageGB‚2709.644561767578 disk.sda.in‚ 9.19140625! disk.sdb.out‚1570.28076171875$ disk.nvme0n1.out‚126307.75390625 disk.md0.in‚ 13.58203125$ memory_percent‚92.05434322357178 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1685.171875 disk.sda2.in‚ 9.19140625# disk.nvme1n1.out‚65601.04296875 disk.rbd2.out‚ 4.095703125 proc.cpu.threads‚17 network.recv‚ 4.529223e+06" disk.sdb2.out‚1570.28076171875 disk.rbd0.out‚ 3.87109375 disk.dm-0.out‚ 191908.796875* proc.memory.percent‚18.387436866760254) disk./.usagePercent‚75.76279602137444 disk.sdb1.out‚0 disk.md0p1.out‚ 1510.69921875Õ =J“´/‚(:ˆ ¼þéÔðê³k gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.047' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.51175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñȸîtµ/‚(:é ¼þéÔà­l disk.md0.out‚ 1512.1953125 disk.sdb2.in‚4.390625( disk./.usagePercent‚75.7629828906267 disk.sdb1.out‚0 disk.md0p1.in‚ 13.58203125 disk.md0.in‚ 13.58203125 disk.sda.in‚ 9.19140625$ memory_percent‚92.05436706542969% disk./.usageGB‚2709.6512451171875 disk.sda1.in‚0 disk.sdb.in‚4.390625$ disk.nvme0n1.out‚126378.51171875* proc.memory.percent‚18.387436866760254 disk.sdb1.in‚0! disk.sda2.out‚1571.8447265625 disk.rbd2.in‚0% proc.memory.availableMB‚ 1301.8125 disk.sda1.out‚0 disk.rbd0.out‚ 3.87109375 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.md0p1.out‚ 1512.1953125 disk.nvme0n1.in‚ 940.51171875# disk.nvme1n1.out‚65604.23046875 disk.sdb.out‚1571.8447265625 disk.sda2.in‚ 9.19140625! disk.sdb2.out‚1571.8447265625# proc.memory.rssMB‚ 3012.59765625 disk.dm-0.out‚191982.7421875 disk.rbd2.out‚ 4.095703125 disk.sda.out‚1571.8447265625 cpu‚25.201330723107453 network.sent‚ 3.1749559e+07 network.recv‚ 4.53392e+06 disk.dm-0.in‚ 1685.171875 disk.nvme1n1.in‚ 744.66015625lP“¶/‚(:ˆ ËþéÔöÛòk gpu.0.gpu‚41.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚37.0 gpu.0.powerWatts‚133.349' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.33725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xMN+|·/‚(:ñ ËþéÔ§´ål disk.sda.in‚ 9.19140625 disk.rbd0.in‚0! disk.sdb.out‚1573.80908203125$ memory_percent‚92.05358028411865 proc.cpu.threads‚17 network.sent‚ 3.1755381e+07 disk.md0p1.in‚ 13.58203125 disk.nvme1n1.in‚ 744.66015625* proc.memory.percent‚18.387436866760254 cpu‚25.201132783004898 disk.md0p1.out‚ 1514.08203125! disk.dm-0.out‚192023.80078125 disk.md0.in‚ 13.58203125 disk.sdb2.in‚4.390625" disk.sdb2.out‚1573.80908203125# proc.memory.rssMB‚ 3012.59765625% disk./.usageGB‚2709.7763175964355 disk.sda1.out‚0 disk.sda2.in‚ 9.19140625 disk.dm-0.in‚ 1685.171875" disk.sda2.out‚1573.80908203125 disk.nvme0n1.in‚ 940.51171875# disk.nvme1n1.out‚65640.67578125 disk.md0.out‚ 1514.08203125 disk.rbd2.in‚0 disk.rbd2.out‚ 4.095703125! disk.sda.out‚1573.80908203125 disk.sdb1.in‚0 disk.sdb1.out‚0 network.recv‚ 4.538088e+06) disk./.usagePercent‚75.76647996949339 disk.sda1.in‚0 disk.rbd0.out‚ 3.87109375) proc.memory.availableMB‚ 1301.94140625 disk.sdb.in‚4.390625 disk.nvme0n1.out‚ 126383.125&Áòâ4¸// _wandbruntime‚14090  _runtime‚14090²ÛÍɹ/à & lr-AdamW/pg1‚7.998721053228144e-07 & lr-AdamW/pg2‚7.998721053228144e-07 trainer/global_step‚11899 % _timestamp‚1.7885100324916658e+09 _runtime‚14090.269024026 _step‚975šŸÎº/È _step‚975 & lr-AdamW/pg1‚7.998721053228144e-07 & lr-AdamW/pg2‚7.998721053228144e-07 trainer/global_step‚11899 % _timestamp‚1.7885100324916658e+09 _runtime‚14090.269024026ÏÒÓÀg4»// _wandbruntime‚14090  _runtime‚14090;Z߇¼/ŽP KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 221073643 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13627248585216 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 248689686 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚11899 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚968922811328752 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 _runtime‚14090.270466283 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 301291121 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218296320 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 248689686 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 149002204 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1002056894378496 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1105610576369152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1002056912381440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1002056894378496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚968922828368112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1015683232992256 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13575389142113 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 301291121 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 550007151 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 149002202 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13626338613760 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 301317465 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚72071439 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1015683232992256 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13627248585216 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 248689686 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 149002204 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1002056894378496 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1105610576369152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1002056912381440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1002056894378496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚968922828368112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1015683232992256 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13575389142113 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 301291121 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 550007151 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 149002202 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13626338613760 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 301317465 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚72071439 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1015683232992256 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13627248585216 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 550025796 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72076536 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1002138531467264 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1015765973538304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1119925317615616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 248709846 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13628370058752 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1002138531467264 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 235931648 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14224727533568 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚982579260353259 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 trainer/global_step‚11899 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 248709846 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 149014269 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1119925553547264 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1105700826013696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚977 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14224515542016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13577414390603 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 550052168 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13628370058752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚969001845962656 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 301342322 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 301315952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 301315952 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚969001828923296 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 550052168 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 221090803 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚631 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1105700802073600 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 _runtime‚14091.274432916 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 211991552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚982578335338687 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1002138513464320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 149014267 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72077167 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1015766901526016 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13576506415391 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13627460073984 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 $ _timestamp‚1.788510033496097e+09gúY›æÀ/œN H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 550052168 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13628370058752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚969001845962656 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 301342322 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 301315952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 301315952 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚969001828923296 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 550052168 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 221090803 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚631 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1105700802073600 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 _runtime‚14091.274432916 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 211991552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚982578335338687 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1002138513464320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 149014267 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72077167 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1015766901526016 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13576506415391 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13627460073984 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 $ _timestamp‚1.788510033496097e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1015766901526016 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 550025796 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 550025796 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72076536 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1002138531467264 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1015765973538304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1119925317615616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 248709846 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13628370058752 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1002138531467264 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 235931648 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14224727533568 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚982579260353259 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 trainer/global_step‚11899 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 248709846 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 149014269 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1119925553547264 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1105700826013696 XJDeviceStats5‡Áö<Monitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚977 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14224515542016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13577414390603ÑŸN4Á// _wandbruntime‚14091  _runtime‚14091G‘Y_Â/Ù .loss/FRt4/detection_loss‚3.034390458693871 1loss/SRqqt2/detection_loss‚1.9641759935766459 1loss/SRqqt3/detection_loss‚0.8556561028692594 .loss/FBt4/detection_loss‚2.371436749513333 _runtime‚14091.275593071 0loss/FBt2/assignment_loss‚0.3489513296603036 /loss/FRt2/detection_loss‚2.5660936248463555 .loss/FBt3/detection_loss‚2.230453485250473 %loss/total_loss‚39.03434369163415 trainer/global_step‚11899 % _timestamp‚1.7885100334983656e+09 /loss/FRt3/detection_loss‚1.6926488970676739 1loss/SRqqt4/detection_loss‚2.3470083991686503 /loss/FBt1/detection_loss‚0.6676983272012037 .loss/FBt2/detection_loss‚1.894333783785502 _step‚978 0loss/FRt4/assignment_loss‚2.9542728456167073 2loss/SRqqt2/assignment_loss‚1.7792682271683589 1loss/FBt4/assignment_loss‚0.20094790596655143 epoch‚9 2loss/SRqqt3/assignment_loss‚0.8532198653171562 1loss/FBt1/assignment_loss‚0.11500155017172595 0loss/FBt3/assignment_loss‚0.0998475992702879 0loss/FRt2/assignment_loss‚2.9954387239507727 2loss/SRqqt4/assignment_loss‚2.1968228795876104 0loss/FRt3/assignment_loss‚1.2617304382785675 /loss/FRt1/detection_loss‚1.5551205738050027 0loss/FRt1/assignment_loss‚1.7816909040723528 1loss/SRqqt1/detection_loss‚1.9357923259958625 2loss/SRqqt1/assignment_loss‚1.3323427007999271Pe$ŒdÃ/Þ 2loss/SRqqt1/assignment_loss‚1.3323427007999271 .loss/FRt4/detection_loss‚3.034390458693871 1loss/SRqqt2/detection_loss‚1.9641759935766459 1loss/SRqqt3/detection_loss‚0.8556561028692594 .loss/FBt4/detection_loss‚2.371436749513333 _runtime‚14091.275593071 0loss/FBt2/assignment_loss‚0.3489513296603036 /loss/FRt2/detection_loss‚2.5660936248463555 .loss/FBt3/detection_loss‚2.230453485250473 %loss/total_loss‚39.03434369163415 trainer/global_step‚11899 % _timestamp‚1.7885100334983656e+09 /loss/FRt3/detection_loss‚1.6926488970676739 1loss/SRqqt4/detection_loss‚2.3470083991686503 /loss/FBt1/detection_loss‚0.6676983272012037 .loss/FBt2/detection_loss‚1.894333783785502 _step‚978 0loss/FRt4/assignment_loss‚2.9542728456167073 2loss/SRqqt2/assignment_loss‚1.7792682271683589 1loss/FBt4/assignment_loss‚0.20094790596655143 epoch‚9 2loss/SRqqt3/assignment_loss‚0.8532198653171562 1loss/FBt1/assignment_loss‚0.11500155017172595 0loss/FBt3/assignment_loss‚0.0998475992702879 0loss/FRt2/assignment_loss‚2.9954387239507727 2loss/SRqqt4/assignment_loss‚2.1968228795876104 0loss/FRt3/assignment_loss‚1.2617304382785675 /loss/FRt1/detection_loss‚1.5551205738050027 0loss/FRt1/assignment_loss‚1.7816909040723528 1loss/SRqqt1/detection_loss‚1.9357923259958625Ò“©ô‘Ä/‚(:† ÚþéÔ¦‚èk gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.512' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹aÛ®tÅ/‚(:é ÚþéÔÁ²äm network.sent‚ 3.176201e+07 disk.sdb1.out‚0 disk.md0p1.in‚ 13.58203125 disk.dm-0.in‚ 1685.171875 disk.nvme1n1.in‚ 744.66015625# disk.nvme1n1.out‚65784.65234375 network.recv‚ 4.543429e+06 disk.sda.in‚ 9.19140625 disk.sda1.out‚0 disk.sda2.in‚ 9.19140625 disk.md0.in‚ 13.58203125 disk.rbd2.in‚0# proc.memory.rssMB‚ 3012.59765625 disk.sdb2.in‚4.390625 cpu‚25.200915532518767 disk.sdb2.out‚1575.302734375 disk.sda.out‚1575.302734375 disk.sdb.out‚1575.302734375 disk.sda2.out‚1575.302734375 disk.rbd2.out‚ 4.095703125$ memory_percent‚92.05536842346191 disk.md0.out‚ 1515.5234375% disk./.usageGB‚2709.9636726379395 disk.rbd0.in‚0 disk.rbd0.out‚ 3.87109375# disk.nvme0n1.out‚126383.1328125( proc.memory.availableMB‚ 1301.6484375) disk./.usagePercent‚75.77171849486801 disk.nvme0n1.in‚ 940.51171875 disk.sda1.in‚0 disk.sdb1.in‚0 disk.md0p1.out‚ 1515.5234375! disk.dm-0.out‚192167.78515625 disk.sdb.in‚4.390625* proc.memory.percent‚18.387436866760254 proc.cpu.threads‚17^'“Æ/‚(:ˆ éþéÔÜòØl gpu.0.gpu‚24.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.763' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.44075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝÁ›tsÇ/‚(:è éþéÔ²ëm disk.md0p1.out‚ 1517.35546875 network.sent‚ 3.1869027e+07 disk.sda1.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚ 13.58203125' proc.memory.availableMB‚ 1301.828125# proc.memory.rssMB‚ 3012.59765625! disk.dm-0.out‚192218.15234375 disk.sdb.in‚4.390625 disk.sda2.out‚1577.208984375$ disk.nvme0n1.out‚126433.26171875) disk./.usagePercent‚75.78399800194958 disk.md0.in‚ 13.58203125 disk.rbd2.in‚0 disk.sdb2.in‚4.390625 disk.md0.out‚ 1517.35546875 disk.sdb2.out‚1577.208984375 disk.rbd0.out‚ 3.87109375 disk.dm-0.in‚ 1685.171875 disk.nvme1n1.in‚ 744.66015625 cpu‚25.20074026546174 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.nvme0n1.in‚ 940.51171875! disk.nvme1n1.out‚ 65784.890625$ memory_percent‚92.05427169799805* proc.memory.percent‚18.387436866760254 proc.cpu.threads‚17 disk.sda2.in‚ 9.19140625 disk.sda.out‚1577.208984375 network.recv‚ 4.548275e+06$ disk./.usageGB‚2710.402847290039 disk.sda.in‚ 9.19140625 disk.sdb.out‚1577.208984375 disk.rbd2.out‚ 4.095703125þktb‘È/‚(:† øþéÔéï©m gpu.0.gpu‚28.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚161.864' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾BtútÉ/‚(:é øþéÔ°°õn disk.sdb2.in‚4.390625 disk.sdb1.in‚0 disk.rbd0.out‚ 3.87109375 disk.sdb1.out‚0 cpu‚25.200556162056042! disk.sdb.out‚1579.23486328125" disk.nvme1n1.out‚ 65928.8515625 disk.sda.in‚ 9.19140625 disk.sda1.out‚0 disk.dm-0.in‚ 1685.171875$ memory_percent‚92.05272197723389 network.recv‚ 4.553118e+06) disk./.usagePercent‚75.78394605827044" disk.sda2.out‚1579.23486328125 disk.nvme0n1.in‚ 940.51171875" disk.sdb2.out‚1579.23486328125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.md0p1.out‚ 1519.3125# proc.memory.rssMB‚ 3012.59765625 proc.cpu.threads‚17 network.sent‚ 3.1877296e+07 disk.sda2.in‚ 9.19140625# disk.nvme0n1.out‚126583.2421875 disk.md0.in‚ 13.58203125 disk.rbd2.out‚ 4.095703125! disk.sda.out‚1579.23486328125 disk.md0p1.in‚ 13.58203125) proc.memory.availableMB‚ 1302.08203125% disk./.usageGB‚2710.4009895324707 disk.sdb.in‚4.390625 disk.md0.out‚ 1519.3125* proc.memory.percent‚18.387436866760254 disk.nvme1n1.in‚ 744.66015625 disk.dm-0.out‚ 192512.09375 disk.rbd2.in‚0™ñÄC“Ê/‚(:ˆ ‡ÿéÔóÖøk gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚157.977' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.49425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_¼µgsË/‚(:è ‡ÿéÔÅȹm# proc.memory.rssMB‚ 3012.59765625* proc.memory.percent‚18.387436866760254 network.sent‚ 3.1880897e+07% disk./.usageGB‚2710.6509895324707 disk.rbd2.out‚ 4.095703125 disk.sda.in‚ 9.19140625 disk.sda.out‚1580.9267578125 disk.sda2.in‚ 9.19140625 disk.nvme1n1.in‚ 744.66015625 disk.rbd2.in‚0! disk.sdb2.out‚1580.9267578125 disk.sdb1.in‚0! disk.nvme1n1.out‚ 66000.921875 disk.rbd0.out‚ 3.87109375 network.recv‚ 4.557503e+06 disk.md0p1.out‚ 1520.9375! disk.dm-0.out‚192684.26171875 disk.sdb.out‚1580.9267578125 disk.rbd0.in‚0 disk.nvme0n1.in‚ 940.51171875 disk.md0.out‚ 1520.9375) proc.memory.availableMB‚ 1301.97265625 disk.sdb2.in‚4.390625 disk.md0p1.in‚ 13.58203125 cpu‚25.200366945948158 disk.sda1.in‚0 disk.dm-0.in‚ 1685.171875! disk.sda2.out‚1580.9267578125 disk.sda1.out‚0) disk./.usagePercent‚75.79093616290356 disk.sdb.in‚4.390625 disk.md0.in‚ 13.58203125$ memory_percent‚92.05338954925537 proc.cpu.threads‚17 disk.sdb1.out‚0$ disk.nvme0n1.out‚126683.33984375œ:q?4Ì// _wandbruntime‚14147  _runtime‚14147š(ÉŠÉÍ/à _step‚979 & lr-AdamW/pg1‚4.359554640945061e-07 & lr-AdamW/pg2‚4.359554640945061e-07 trainer/global_step‚11949 % _timestamp‚1.7885100898930767e+09 _runtime‚14147.670325525’Û§ÎÎ/È trainer/global_step‚11949 % _timestamp‚1.7885100898930767e+09 _runtime‚14147.670325525 _step‚979 & lr-AdamW/pg1‚4.359554640945061e-07 & lr-AdamW/pg2‚4.359554640945061e-07Óh(Cº4Ï// _wandbruntime‚14147  _runtime‚14147F÷̤(Ð/ŽP ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚986504957453173 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 552233707 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 302562367 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1110121885531648 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1006137866672640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _runtime‚14147.671265579 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 249697684 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1110121861591552 _step‚980 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 249697686 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1124403730440704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚645 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚972873691055952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚72362280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13632174372225 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 249697684 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 302562367 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚647 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1019820274708480 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1019820274708480 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚645 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚972873691055952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚72362280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13632174372225 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 249697684 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 302562367 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚647 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚981 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1006219485758464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚635 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocatedøµES$‚615 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13684457483776 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 249717846 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14283064702976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13632383670499 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 552278698 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚972952708650496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 221985612 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚972952725689856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 249717844 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1006219467755520 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13683547499008 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1110212111236096 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13633291645711 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 249717846 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1019903015254528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 221984977 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 149617519 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 552305070 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233834496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13683547499008 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚986586017335567 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 552278698 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 249717844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 552305070 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚633 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72367460 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1006219485758464 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 302587224 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1110212087296000 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 % _timestamp‚1.7885100908628743e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 552305070 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚633 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72367460 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1006219485758464 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 302587224 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1110212087296000 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 % _timestamp‚1.7885100908628743e+09 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚981 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1006219485758464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚635 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13684457483776 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 249717846 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14283064702976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13632383670499 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 552278698 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚972952708650496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 221985612 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚972952725689856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 249717844 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1006219467755520 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13683547499008 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1110212111236096 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13633291645711 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 249717846 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1019903015254528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 221984977 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 149617519 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 552305070 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233834496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13683547499008 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚986586017335567 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 552278698 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 249717844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13738513463808 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1128974063380992 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1114633170754048 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 150208702 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14340892626944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 554486609 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 250705686 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚72654342 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1023958244399104 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14340668039168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚990512639449845 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1023957316424704 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1010218820963840 E;DeviceStatË"z(sMonitor.on_train_batch_start/allocation.all.peak‚32249 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 554512955 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1128973814853120 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1010218802960896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚662 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚990511714435481 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 303807269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13688051627333 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13738513463808 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 303780925 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 _runtime‚14206.281818321 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13739423435264 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 150208704 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚976824570783152 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1010218820963840 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 222863044 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 trainer/global_step‚11999 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 303780925 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 222863708 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1023957316424704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1023958244399104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚72655004 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 250705684 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 303807269 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 250705684 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144F‡5ë(å/“P \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13739423435264 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1128974063380992 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1114633170754048 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 150208702 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14340892626944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 554486609 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 250705686 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚72654342 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1023958244399104 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14340668039168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚990512639449845 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1023957316424704 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1010218820963840 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 554512955 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1128973814853120 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224587776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1010218802960896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚662 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚990511714435481 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 303807269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13688051627333 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13738513463808 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 303780925 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248527872 _runtime‚14206.281818321 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13739423435264 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 150208704 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚976824570783152 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1010218820963840 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 222863044 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 trainer/global_step‚11999 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 303780925 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 222863708 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1023957316424704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1023958244399104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚72655004 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 250705684 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 303807269 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 250705684 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13738513463808 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885101497231631e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13739634924032 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1024040984958464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 222880771 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚976903588377696 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1114723372518400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1010300440049664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 554531600 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚976903605417056 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 222881433 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1129065462301696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14341834977280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1024040984958464 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1129065207495680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13688260925607 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 554557972 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 554557972 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1024040056970752 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 150220767 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚660 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚662 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚14207.501794116 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1010300440049664 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 250725844 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 303832126 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 303805756 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72660664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 250725846 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 250725846 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 554531600 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1010300422046720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 303805756 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1010300422046720 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚990591849303303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72660004 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14342065843200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 250725844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 303832126 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13740544908800 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885101497231631e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13739634924032 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1024040984958464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 222880771 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚976903588377696 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1114723372518400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1010300440049664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 554531600 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚976903605417056 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 222881433 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1129065462301696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14341834977280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1024040984958464 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1129065207495680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230865920 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13688260925607 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 554557972 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 554557972 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1024040056970752 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 150220767 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚660 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚662 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 355286016 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚14207.501794116 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1010300440049664 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 2507258~„F°Ù44 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 303832126 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 303805756 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72660664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 250725846 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 250725846 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 554531600 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1010300422046720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 303805756 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1010300422046720 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚990591849303303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72660004 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14342065843200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 250725844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 303832126 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13740544908800 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0ÙWÙÐ 4é// _wandbruntime‚14207  _runtime‚14207IœýZê/Ô .loss/FBt2/detection_loss‚2.647883834110366 trainer/global_step‚11999 % _timestamp‚1.7885101497259784e+09 /loss/FRt1/assignment_loss‚2.433931763579206 1loss/FBt1/assignment_loss‚0.04409194687779063 1loss/SRqqt4/detection_loss‚2.5847731232643127 _step‚986 0loss/FRt2/assignment_loss‚2.9757258365320607 1loss/FBt2/assignment_loss‚0.09013766614513467 /loss/FBt3/detection_loss‚1.9605756203333538 epoch‚9 _runtime‚14207.503384903 1loss/SRqqt2/assignment_loss‚1.296440309935039 2loss/SRqqt4/assignment_loss‚1.3067504254480202 1loss/FBt4/assignment_loss‚0.37579278735265687 /loss/FRt3/detection_loss‚1.7504645455955414 /loss/FBt1/detection_loss‚0.8296608585229313 /loss/FBt4/detection_loss‚2.4043863356113433 0loss/FRt4/assignment_loss‚3.3430092388933357 1loss/SRqqt1/detection_loss‚2.1517002341838984 2loss/SRqqt3/assignment_loss‚0.9120390132635033 0loss/SRqqt2/detection_loss‚2.368744206043982 $loss/total_loss‚40.8535178885064 1loss/SRqqt3/detection_loss‚0.8988649783774264 0loss/FBt3/assignment_loss‚0.2245513519931895 .loss/FRt2/detection_loss‚2.156796002717212 /loss/FRt3/assignment_loss‚1.330400902973978 2loss/SRqqt1/assignment_loss‚1.5421715949017267 ,loss/FRt1/detection_loss‚1.8541543003418 /loss/FRt4/detection_loss‚3.3704710115085947¢Ëz£_ë/Ù trainer/global_step‚11999 % _timestamp‚1.7885101497259784e+09 /loss/FRt1/assignment_loss‚2.433931763579206 1loss/FBt1/assignment_loss‚0.04409194687779063 1loss/SRqqt4/detection_loss‚2.5847731232643127 _step‚986 0loss/FRt2/assignment_loss‚2.9757258365320607 1loss/FBt2/assignment_loss‚0.09013766614513467 /loss/FBt3/detection_loss‚1.9605756203333538 epoch‚9 _runtime‚14207.503384903 1loss/SRqqt2/assignment_loss‚1.296440309935039 2loss/SRqqt4/assignment_loss‚1.3067504254480202 1loss/FBt4/assignment_loss‚0.37579278735265687 /loss/FRt3/detection_loss‚1.7504645455955414 /loss/FBt1/detection_loss‚0.8296608585229313 /loss/FBt4/detection_loss‚2.4043863356113433 0loss/FRt4/assignment_loss‚3.3430092388933357 1loss/SRqqt1/detection_loss‚2.1517002341838984 2loss/SRqqt3/assignment_loss‚0.9120390132635033 0loss/SRqqt2/detection_loss‚2.368744206043982 $loss/total_loss‚40.8535178885064 1loss/SRqqt3/detection_loss‚0.8988649783774264 0loss/FBt3/assignment_loss‚0.2245513519931895 .loss/FRt2/detection_loss‚2.156796002717212 /loss/FRt3/assignment_loss‚1.330400902973978 2loss/SRqqt1/assignment_loss‚1.5421715949017267 ,loss/FRt1/detection_loss‚1.8541543003418 /loss/FRt4/detection_loss‚3.3704710115085947 .loss/FBt2/detection_loss‚2.647883834110366ÚœÄÙa’ì/‚(:‡ ÒÿéÔ—ÕÈn gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.774' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.9435 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶s©£jí/‚(:ß ÒÿéÔÊùéo disk.sda1.out‚0) proc.memory.availableMB‚ 1301.56640625 cpu‚25.19933838800079 proc.cpu.threads‚17 disk.sdb1.out‚0$ memory_percent‚92.05586910247803 disk.sdb1.in‚0 network.recv‚ 4.580629e+06 disk.md0.in‚ 13.58203125$ disk./.usageGB‚2710.121536254883 disk.sda2.out‚1589.537109375 disk.rbd2.in‚0 disk.sdb.out‚1589.537109375! disk.nvme1n1.out‚ 66296.828125 disk.rbd2.out‚ 4.095703125 disk.dm-0.out‚ 193431.078125 network.sent‚ 3.2012547e+07 disk.md0p1.in‚ 13.58203125 disk.rbd0.out‚ 3.87109375 disk.md0p1.out‚ 1529.2265625( disk./.usagePercent‚75.7761324276688 disk.sdb.in‚4.390625 disk.sda2.in‚ 9.19140625 disk.nvme0n1.in‚ 940.51171875 disk.sdb2.in‚4.390625 disk.sda.in‚ 9.19140625 disk.nvme0n1.out‚ 127134.25 disk.sdb2.out‚1589.537109375 disk.sda.out‚1589.537109375 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1685.171875# proc.memory.rssMB‚ 3012.59765625* proc.memory.percent‚18.387436866760254 disk.nvme1n1.in‚ 744.66015625 disk.md0.out‚ 1529.2265625wI>Š’î/‚(:‡ áÿéÔ÷øŒm gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.422' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.8555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿK~~ï/‚(:ó áÿéÔäàÓn network.recv‚ 4.585615e+06) disk./.usagePercent‚75.77738099585781 disk.nvme0n1.in‚ 940.51171875) proc.memory.availableMB‚ 1301.44921875 disk.nvme1n1.in‚ 744.66015625 disk.md0.out‚ 1530.84765625 disk.sda1.in‚0$ memory_percent‚92.05658435821533 cpu‚25.19908044203442$ disk.nvme0n1.out‚127284.32421875 disk.rbd0.in‚0$ disk./.usageGB‚2710.166191101074 disk.rbd2.out‚ 4.095703125 disk.sda.in‚ 9.19140625! disk.sda.out‚1591.22216796875 disk.rbd0.out‚ 3.87109375 disk.sdb1.in‚0 disk.md0p1.out‚ 1530.84765625 disk.sdb1.out‚0 network.sent‚ 3.2121861e+07! disk.dm-0.out‚193673.44140625 disk.dm-0.in‚ 1685.171875! disk.sdb.out‚1591.22216796875 disk.sda2.in‚ 9.19140625 disk.md0.in‚ 13.58203125" disk.sdb2.out‚1591.22216796875 disk.sda1.out‚0# proc.memory.rssMB‚ 3012.59765625 proc.cpu.threads‚17 disk.md0p1.in‚ 13.58203125 disk.sdb.in‚4.390625" disk.sda2.out‚1591.22216796875" disk.nvme1n1.out‚ 66389.1171875 disk.rbd2.in‚0 disk.sdb2.in‚4.390625* proc.memory.percent‚18.387436866760254帳 ’ð/‚(:‡ ðÿéÔÃý­m gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{¹x|ñ/‚(:ñ ðÿéÔ”Óên disk.sda1.out‚0 disk.rbd0.out‚ 3.87109375 disk.md0p1.out‚ 1532.38671875( proc.memory.availableMB‚ 1301.5859375 network.sent‚ 3.2127923e+07 disk.dm-0.in‚ 1685.171875 disk.sdb.in‚4.390625 disk.md0.in‚ 13.58203125# proc.memory.rssMB‚ 3012.59765625 cpu‚25.19883553518589 network.recv‚ 4.59022e+06) disk./.usagePercent‚75.77921630363403 disk.rbd2.in‚0 disk.sdb1.in‚0" disk.sdb2.out‚1592.82568359375 disk.sda1.in‚0* proc.memory.percent‚18.387436866760254 proc.cpu.threads‚17 disk.sda.in‚ 9.19140625 disk.nvme0n1.in‚ 940.51171875 disk.nvme1n1.in‚ 744.66015625 disk.md0.out‚ 1532.38671875 disk.sdb1.out‚0 disk.md0p1.in‚ 13.58203125 disk.sda2.in‚ 9.19140625# disk.nvme1n1.out‚66392.29296875 disk.rbd2.out‚ 4.095703125! disk.dm-0.out‚193772.64453125! disk.sdb.out‚1592.82568359375# disk.nvme0n1.out‚127380.3515625$ memory_percent‚92.05574989318848! disk.sda.out‚1592.82568359375 disk.rbd0.in‚0" disk.sda2.out‚1592.82568359375$ disk./.usageGB‚2710.231830596924 disk.sdb2.in‚4.390625[v’ò/‚(:‡ ÿÿéÔËß×m gpu.0.gpu‚17.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.774' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.9435 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05Á« {ó/‚(:ð ÿÿéԸ؈o disk.dm-0.in‚ 1728.1328125" disk.sdb2.out‚1594.77099609375$ memory_percent‚92.05465316772461) disk./.usagePercent‚75.78556260529527! disk.dm-0.out‚193889.26171875 disk.sda1.in‚0 cpu‚25.19860844218085 network.sent‚ 3.2133743e+07% disk./.usageGB‚2710.4588050842285" disk.nvme0n1.out‚ 127386.796875 disk.md0.out‚ 1534.2734375! disk.sda.out‚1594.77099609375' proc.memory.availableMB‚ 1301.765625 network.recv‚ 4.593875e+06 disk.nvme0n1.in‚ 979.54296875 disk.rbd2.out‚ 4.095703125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0p1.out‚ 1534.2734375* proc.memory.percent‚18.387436866760254" disk.sda2.out‚1594.77099609375# disk.nvme1n1.out‚66502.46484375 disk.sda.in‚ 9.19140625# proc.memory.rssMB‚ 3012.59765625 disk.rbd0.out‚ 3.87109375 disk.sda2.in‚ 9.19140625 disk.nvme1n1.in‚ 748.58984375 disk.md0.in‚ 13.58203125 disk.sdb2.in‚4.390625 disk.sda1.out‚0 disk.sdb.in‚4.390625! disk.sdb.out‚1594.77099609375 disk.rbd2.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.md0p1.in‚ 13.58203125*—êŽ4ô// _wandbruntime‚14267  _runtime‚14267:‹˜†Éõ/à _step‚987 & lr-AdamW/pg1‚3.686604275170815e-08 & lr-AdamW/pg2‚3.686604275170815e-08 trainer/global_step‚12049 % _timestamp‚1.7885102095781696e+09 _runtime‚14267.355513321ËÈ%šÎö/È _runtime‚14267.355513321 _step‚987 & lr-AdamW/pg1‚3.686604275170815e-08 & lr-AdamW/pg2‚3.686604275170815e-08 trainer/global_step‚12049 % _timestamp‚1.7885102095781696e+09ÛÞ…Ø 4÷// _wandbruntime‚14267  _runtime‚14267C@9(ø/ŽP QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 251713686 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1119144455976448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚994518471417789 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1028094358140928 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚994519396432153 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚72946359 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570\É6‡‚"ù/“P D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13795510860288 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚994519396432153 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚72946359 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 251713686 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1119144455976448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚994518471417789 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1028094358140928 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 556810874 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚653 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72952724 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 556784502 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚14268.567982951 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 150824019 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1028178026674688 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13795722349056 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 150824017 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 223776743 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13745046155927 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 251733844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚980854468104896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1133635420165120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1028177098686976 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚651 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 251733846 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 305050658 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13796632333824 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 556784502 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 305050658 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _step‚989 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13795722349056 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1028178026674688 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 223776090 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14400738484224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1014381376337920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13796632333824 trainer/global_step‚12049 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 305077028 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13796632333824 trainer/global_step‚12049 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 305077028 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 556810874 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚653 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72952724 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 556784502 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚14268.567982951 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 150824019 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1028178026674688 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13795722349056 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 150824017 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 223776743 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13745046155927 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 251733844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚980854468104896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1133635420165120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1028177098686976 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚651 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 251733846 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 305050658 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13796632333824 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 556784502 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 305050658 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _step‚989 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13795722349056 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1028178026674688 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 223776090 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14400738484224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1014381376337920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚615 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚615 % _timestamp‚1.7885103162453678e+09 gRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚1125600796054528 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚14471861869056 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 151775157 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1195376640 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14401142784 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1185 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚1034169758141440 _JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚1020304924214272 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚239 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 701042176 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚986486068080032 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 559781329 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 683039232 [FDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚1034169758141440 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20711 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1187728393 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 559802042 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚615 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚615 % _timestamp‚1.7885103162453678e+09 gRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚1125600796054528 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚14471861869056 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 151775157 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1195376640 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14401142784 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1185 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚1034169758141440 _JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚1020304924214272 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚239 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 355286016 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 701042176 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚986486068080032 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 559781329 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 683039232 [FDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚1034169758141440 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20711 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1187728393 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 559802042 FJÆ4‘0/ _wandbruntime‚14375  _runtime‚14375©Mg’¨ ’0S ZEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚1034225209941504 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 559805619 aLDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚1020359203124224 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 14135738368 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 306512286 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 306512286 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1195376640 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚570 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32006 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚13866006817280 YDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚1034224508899328 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135738368 ^IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚1140129221757952 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚986536477145856 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 trainer/global_step‚12089 bMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚1140129439268352 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 % _timestamp‚1.7885103172979162e+09 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 306491575 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 225232710 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 683039232 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189768192 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 701042176 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚14472158149632 VADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚1034224508899328 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚13865323778048 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 dODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚1020359203124224 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3725 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 409073152 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 253314044 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚615 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 683039232 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 PIDeviceStatsMonitor.on_validaŠ ‡ßtion_batch_end/inactive_split.large_pool.peak‚44 iTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚1125657087548416 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚13814172067947 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 253314044 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚14472351719936 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 701042176 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 559826332 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 193570304 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 225232469 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚239 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚1000350666253163 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚241 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 681067448 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚13813491000499 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13684416521 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14401142784 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚615æ=aõŒ)“0†S RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13684416521 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14401142784 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚615 ZEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚1034225209941504 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1187728393 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 559805619 aLDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚1020359203124224 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 14135738368 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 306512286 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 306512286 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1195376640 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚570 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32006 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚13866006817280 YDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚1034224508899328 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135738368 ^IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚1140129221757952 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚986536477145856 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 trainer/global_step‚12089 bMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚1140129439268352 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 % _timestamp‚1.7885103172979162e+09 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 306491575 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 225232710 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 683039232 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189768192 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 701042176 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚14472158149632 VADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚1034224508899328 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚13865323778048 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 dODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚1020359203124224 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3725 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 409073152 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 253314044 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚615 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 683039232 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚44 iTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚1125657087548416 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚13814172067947 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 253314044 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚14472351719936 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 701042176 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 559826332 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 193570304 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 225232469 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚239 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚1000350666253163 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚241 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 681067448 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚13813491000499à/Ž1Å‘”0‚(:† ÷€êÔÕîÜl gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.408218383789062* gpu.0.memoryAllocatedBytes‚ 14953545728 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.3665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çïÊ3w•0‚(:ì ÷€êԟ̉m disk.md0p1.in‚ 13.58203125 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 1748.38671875 disk.rbd2.in‚0 disk.rbd2.out‚ 4.37451171875 network.sent‚ 3.2279587e+07 disk.sdb1.out‚0# disk.nvme1n1.out‚67035.36328125) disk./.usagePercent‚75.75972729120475# disk.nvme0n1.out‚127996.4609375 disk.sdb2.in‚4.390625! disk.sda.out‚1607.78564453125! disk.sdb.out‚1607.78564453125$ disk./.usageGB‚2709.534809112549 disk.md0p1.out‚ 1546.78125 disk.sdb.in‚4.390625 disk.md0.in‚ 13.58203125! disk.dm-0.out‚195031.82421875" disk.sdb2.out‚1607.78564453125 disk.sda.in‚ 9.19140625 disk.sdb1.in‚0 disk.nvme0n1.in‚ 981.265625 disk.rbd0.out‚ 4.150390625& proc.memory.availableMB‚ 1303.28125* proc.memory.percent‚18.387460708618164" disk.sda2.out‚1607.78564453125 disk.nvme1n1.in‚ 767.12109375 disk.md0.out‚ 1546.78125$ memory_percent‚92.04540252685547" proc.memory.rssMB‚ 3012.6015625 cpu‚25.21734652143227 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚ 9.19140625 network.recv‚ 4.627905e+06 «Þ¸4–0/ _wandbruntime‚14389  _runtime‚14389ÊŸYâ/G—0‘¦ ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.33860456943511963 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4174328148365021 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5961309671401978 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.23310810327529907 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.13312600553035736 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.018865643069148064 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.411874383687973 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9927732944488525 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.44675111770629883 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21096810698509216 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_3_of_7‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9895833134651184 6validation_average_jet_accuracy‚0.4232519567012787 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.24747473001480103 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.993088960647583 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4122343063354492 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5445418357849121 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5340501070022583 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 jet/accuracy_6_of_6‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.385437548160553 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011287803761661053 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.1111963763833046 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 particle/accuracy_1_of_7‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.25 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.34858161211013794 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.25 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0817236676812172 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.578864336013794 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_5_of_7‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6111111044883728 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5267537236213684 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4554184675216675 particle/accuracy_1_of_2‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03121308982372284 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4629629850387573 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 epoch‚9 % _timestamp‚1.7885103317845068e+09 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6125097274780273 +jet/accuracy_3_of_3‚0.02615024894475937 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5349358916282654 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.43870431184768677 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_1_of_7‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.030523553490638733 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07636473327875137 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 trainer/global_step‚12089 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9920712113380432 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4270833432674408 _runtime‚14389.564319382 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4821763038635254 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9973360300064087 *jet/accuracy_1_of_4‚0.9079574346542358 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9913134574890137 8"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.540123462677002 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.18311448395252228 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5251500606536865 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3997162878513336 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7048554420471191 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.43085992336273193 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9946901202201843 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0092398039996624 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.029188714921474457 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.15816013514995575 particle/accuracy_3_of_3‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3541666567325592 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5625 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5215277075767517 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.47960105538368225 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0989583358168602 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5002824664115906 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9916458129882812 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5201645493507385 +particle/sensitivity‚0.3319638669490814 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2916666865348816 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 6 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.165245920419693 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.39931759238243103 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.40119054913520813 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9821428656578064 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5087719559669495 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3737204670906067 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4280303120613098 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.43215492367744446 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9399999976158142 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23786617815494537 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.0859713926911354 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09649123251438141 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.022637944668531418 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.011111111380159855 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2539510726928711 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8106329441070557 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 jet/accuracy_5_of_7‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.45557212829589844 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1715482622385025 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.030122658237814903 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5714285969734192 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7441303730010986 +jet/accuracy_2_of_5‚0.45045438408851624 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.44885361194610596 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1634790599346161 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.27906250953674316 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7121211886405945 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5373563170433044 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.060652706772089005 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2476172149181366 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9166667461395264 particle/accuracy_2_of_6‚1 particle/accuracy_3_of_6‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.46496501564979553 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,jet/accuracy_4_of_5‚0.005376344081014395 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9889770746231079 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.45173609256744385 particle/accuracy_1_of_6‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.17869248986244202 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991071224212646 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07072018831968307 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.45615094900131226 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2142857164144516 +jet/accuracy_3_of_5‚0.08057475835084915 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6309726238250732 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8648899793624878 jet/accuracy_2_of_7‚0.5 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ( Purity/*FRt3SRqqt2FBt/FBt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9985767006874084 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.48973724246025085 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1962268203496933 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5562578439712524 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.43376582860946655 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.25272998213768005 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4789256155490875 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9864341616630554 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20142120122909546 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 & Purity/1FRt1SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2618449032306671 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5522335171699524 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3190859258174896 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1784910410642624 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9878621697425842 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.45335450768470764 jet/accuracy_5_of_5‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9845823645591736 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9629628658294678 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03819425776600838 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/16ÊáéKFRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1862281858921051 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 particle/accuracy_4_of_7‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5353944301605225 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_4‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5625 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2509259283542633 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.5256410241127014 particle/accuracy_3_of_7‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24800901114940643 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_4_of_4‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 *jet/accuracy_2_of_2‚0.1054096594452858 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9722222089767456 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21442857384681702 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_6_of_7‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.18832305073738098 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3165774643421173 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5147988796234131 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 'particle/f_score‚0.4097594916820526 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5686662197113037 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 particle/accuracy_1_of_1‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.13636364042758942 particle/accuracy_4_of_6‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1807376593351364 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4918577969074249 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.43085992336273193 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4558843970298767 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 particle/accuracy_2_of_4‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1942213773727417 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5172099471092224 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.982539713382721 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9407770037651062 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322916865348816 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.18989941477775574 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5799999833106995 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9930245876312256 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4579395651817322 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.44380804896354675 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9722222685813904 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.44117647409439087 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0009920635493472219 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6818181872367859 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.04150943458080292 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5048449039459229 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3991071283817291 *jet/accuracy_1_of_6‚0.9807692170143127 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4629629850387573 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.49091050028800964 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 *jet/accuracy_3_of_6‚0.1666666567325592 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 particle/accuracy_2_of_7‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23935024440288544 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5646929740905762 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0543154776096344 particle/accuracy_2_of_2‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 +jet/accuracy_4_of_6‚0.07692307978868484 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5548762083053589 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4739071726799011 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23202720284461975 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4407055675983429 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9870368838310242 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_5‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5729166865348816 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1671493947505951 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1735106259584427 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4704568684101105 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.5086361169815063 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.39613595604896545 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3472222089767456 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2614942491054535 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3124999701976776 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18482543528079987 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1678362488746643 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6126505732536316 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3481481671333313 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15550057590007782 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2840460538864136 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9684140086174011 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.41333332657814026 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.35122549533843994 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.14559385180473328 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 )loss/val_total_loss‚39.90687083758675 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.45412641763687134 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 particle/accuracy_6_of_6‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02948094718158245 +jet/accuracy_3_of_4‚0.07077700644731522 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3515249788761139 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ( Purity/1FRt3SRqqt2FBt/FBt_purity‚0.5 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 _step‚993 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995235204696655 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8507875204086304 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9349881410598755 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18542450666427612 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.987500011920929 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09670782089233398 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1489361673593521 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9888785481452942 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2504017651081085 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.18591953814029694 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 particle/accuracy_5_of_6‚1 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0018155410652980208 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9922897815704346 5 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.28555828332901 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.17017638683319092 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.020641792565584183 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1688888669013977 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.641865074634552 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.39357179403305054 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10469970107078552 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5203641057014465 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5664135217666626 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4893518388271332 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.38981035351753235 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12795749306678772 ,jet/accuracy_4_of_4‚0.007781208027154207 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4865576922893524 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.5246914029121399 jet/accuracy_7_of_7‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9915949702262878 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 6 Purity/1FRt3SRqqt*FBt/FBt_purity‚0.970588207244873 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 *jet/accuracy_1_of_1‚0.4559958577156067 particle/accuracy_1_of_3‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.35816994309425354 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24724479019641876 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1807376593351364 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.18705050647258759 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2361111342906952 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6000000238418579 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.46463119983673096 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.44924935698509216 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.19530315697193146 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 *particle/specificity‚0.946804404258728 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1547618955373764 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.22717994451522827 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013307593762874603 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4438285827636719 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_7‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4726145565509796 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.3809523582458496 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9915853142738342 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.260166198015213 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.4375 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_4_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4728050231933594 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3502443730831146 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5666666626930237 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18599969148635864 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812NHóy 4˜0–¦ >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,jet/accuracy_4_of_5‚0.005376344081014395 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9889770746231079 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.45173609256744385 particle/accuracy_1_of_6‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.17869248986244202 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9991071224212646 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07072018831968307 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.45615094900131226 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2142857164144516 +jet/accuracy_3_of_5‚0.08057475835084915 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6309726238250732 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8648899793624878 jet/accuracy_2_of_7‚0.5 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ( Purity/*FRt3SRqqt2FBt/FBt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9985767006874084 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.48973724246025085 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1962268203496933 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5562578439712524 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.43376582860946655 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.25272998213768005 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4789256155490875 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9864341616630554 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20142120122909546 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 & Purity/1FRt1SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2618449032306671 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5522335171699524 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3190859258174896 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1784910410642624 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9878621697425842 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.45335450768470764 jet/accuracy_5_of_5‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9845823645591736 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9629628658294678 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03819425776600838 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1862281858921051 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 particle/accuracy_4_of_7‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5353944301605225 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_4‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5625 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2509259283542633 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.5256410241127014 particle/accuracy_3_of_7‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24800901114940643 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_4_of_4‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 *jet/accuracy_2_of_2‚0.1054096594452858 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9722222089767456 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21442857384681702 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_6_of_7‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.18832305073738098 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3165774643421173 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5147988796234131 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 'particle/f_score‚0.4097594916820526 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5686662197113037 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 particle/accuracy_1_of_1‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.13636364042758942 particle/accuracy_4_of_6‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1807376593351364 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4918577969074249 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.43085992336273193 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4558843970298767 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 particle/accuracy_2_of_4‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1942213773727417 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5172099471092224 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.982539713382721 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9407770037651062 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322916865348816 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.18989941477775574 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5799999833106995 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9930245876312256 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4579395651817322 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.44380804896354675 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9722222685813904 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.44117647409439087 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0009920635493472219 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6818181872367859 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.04150943458080292 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5048449039459229 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3991071283817291 *jet/accuracy_1_of_6‚0.9807692170143127 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4629629850387573 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.49091050028800964 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 *jet/accuracy_3_of_6‚0.1666666567325592 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 particle/accuracy_2_of_7‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23935024440288544 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5646929740905762 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0543154776096344 particle/accuracy_2_of_2‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 +jet/accuJ*¬$_racy_4_of_6‚0.07692307978868484 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5548762083053589 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4739071726799011 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23202720284461975 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4407055675983429 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9870368838310242 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_5‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5729166865348816 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1671493947505951 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1735106259584427 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4704568684101105 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.5086361169815063 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.39613595604896545 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3472222089767456 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2614942491054535 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3124999701976776 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18482543528079987 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1678362488746643 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6126505732536316 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3481481671333313 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15550057590007782 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2840460538864136 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9684140086174011 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.41333332657814026 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.35122549533843994 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.14559385180473328 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 )loss/val_total_loss‚39.90687083758675 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.45412641763687134 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 particle/accuracy_6_of_6‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02948094718158245 +jet/accuracy_3_of_4‚0.07077700644731522 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3515249788761139 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ( Purity/1FRt3SRqqt2FBt/FBt_purity‚0.5 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 _step‚993 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995235204696655 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8507875204086304 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9349881410598755 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18542450666427612 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.987500011920929 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09670782089233398 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1489361673593521 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9888785481452942 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2504017651081085 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.18591953814029694 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 particle/accuracy_5_of_6‚1 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0018155410652980208 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9922897815704346 5 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.28555828332901 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.17017638683319092 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.020641792565584183 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1688888669013977 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.641865074634552 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.39357179403305054 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10469970107078552 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5203641057014465 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5664135217666626 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4893518388271332 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.38981035351753235 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12795749306678772 ,jet/accuracy_4_of_4‚0.007781208027154207 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4865576922893524 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.5246914029121399 jet/accuracy_7_of_7‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9915949702262878 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 6 Purity/1FRt3SRqqt*FBt/FBt_purity‚0.970588207244873 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 *jet/accuracy_1_of_1‚0.4559958577156067 particle/accuracy_1_of_3‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.35816994309425354 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24724479019641876 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1807376593351364 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.18705050647258759 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2361111342906952 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6000000238418579 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.46463119983673096 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.44924935698509216 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.19530315697193146 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 *particle/specificity‚0.946804404258728 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1547618955373764 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.22717994451522827 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013307593762874603 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4438285827636719 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_7‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4726145565509796 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.3809523582458496 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9915853142738342 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.260166198015213 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.4375 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_4_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4728050231933594 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3502443730831146 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5666666626930237 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18599969148635864 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.33860456943511963 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4174328148365021 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5961309671401978 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.23310810327529907 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.13312600553035736 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.018865643069148064 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.411874383687973 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9927732944488525 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.44675111770629883 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21096810698509216 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_3_of_7‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9895833134651184 6validation_average_jet_accuracy‚0.4232519567012787 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.24747473001480103 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.993088960647583 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4122343063354492 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5445418357849121 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5340501070022583 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 jet/accuracy_6_of_6‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.385437548160553 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011287803761661053 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.1111963763833046 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 particle/accuracy_1_of_7‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.25 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.34858161211013794 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.25 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0817236676812172 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.578864336013794 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_5_of_7‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6111111044883728 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5267537236213684 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4554184675216675 particle/accuracy_1_of_2‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03121308982372284 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4629629850387573 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 epoch‚9 % _timestamp‚1.7885103317845068e+09 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6125097274780273 +jet/accuracy_3_of_3‚0.02615024894475937 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5349358916282654 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.43870431184768677 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_1_of_7‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.030523553490638733 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07636473327875137 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 trainer/global_step‚12089 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9920712113380432 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4270833432674408 _runtime‚14389.564319382 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4821763038635254 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9973360300064087 *jet/accuracy_1_of_4‚0.9079574346542358 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9913134574890137 8"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.540123462677002 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.18311448395252228 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5251500606536865 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3997162878513336 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7048554420471191 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.43085992336273193 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9946901202201843 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0092398039996624 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.029188714921474457 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.15816013514995575 particle/accuracy_3_of_3‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3541666567325592 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5625 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5215277075767517 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.47960105538368225 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0989583358168602 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5002824664115906 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9916458129882812 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5201645493507385 +particle/sensitivity‚0.3319638669490814 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2916666865348816 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 6 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.165245920419693 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.39931759238243103 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.40119054913520813 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9821428656578064 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5087719559669495 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3737204670906067 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4280303120613098 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.43215492367744446 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9399999976158142 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23786617815494537 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.0859713926911354 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09649123251438141 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.022637944668531418 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.011111111380159855 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2539510726928711 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8106329441070557 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 jet/accuracy_5_of_7‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.45557212829589844 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1715482622385025 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.030122658237814903 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5714285969734192 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7441303730010986 +jet/accuracy_2_of_5‚0.45045438408851624 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.44885361194610596 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1634790599346161 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.27906250953674316 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7121211886405945 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5373563170433044 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.060652706772089005 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2476172149181366 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9166667461395264 particle/accuracy_2_of_6‚1 particle/accuracy_3_of_6‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.46496501564979553á5ÙG!R™0 okrvw1yuj@ û€êÔÏÕ 0`Trainer.fit` stopped: `max_epochs=10` reached. „íˆ4š0/ _wandbruntime‚14392  _runtime‚14392¶åÇ›0‚(’¸p